OK, case solved. I found the deed of manumission for one set of the "by" people - Crippin - (didn't have time to search all of the surnames). They had been freed a couple years before the 1820 census (1818) and I expect that is why they were marked out in the census.
Many thanks for pointing me in the right direction.