I see the listing you mentioned for 1820 on ancestry:

Name: James Banister
Township: Lexington
County: Rockbridge
State: Virginia
Year: 1820
Roll: M33_130
Page: 279
Image Number: 508

I don't see any first name "James" in Rockbridge County 1830, 1840 - that even vaguely could be spelled or misspelled like Bannister.

In 1830 there is this free colored listing"

Name: Bannister, Edward
Township: Not Stated
County: Rockbridge
State: Virginia
Year: 1830
Roll: 199
Page: 291

Has one male 55 to 100, one 10 to 24 and one female 34 to 55

There is this 1840 free colored listing:

Name: A Bannister
Township: Not Stated
County: Rockbridge
State: Virginia
Roll: 577
Page: 200

2 males 10 and under 24, 1 36 and under 55, 3 females under 10, one 10 under 24 and one 36 under 55

Hope this helps,

Denise