Della,
Have you checked the census prior to 1850 for other RICHARDSONs who were Free People of Color in the County? There were RICHARDSONs who were free during that time period in NC & VA and searching the earlier census is one way to see some of the names in the family were living in the area who were also free. In taking a quick look in Halifax Co.,VA, I see a SAMUEL RICHARDSON, FN (Free Negro) listed in 1820 Marseilles Dist., with a household that included 4 males 14-26 years old, 1 male over 45 (probably Samuel), 1 female 26-45 years old, and 1 female over 45.
You don't mention Raleigh RICHARDSON's age, but it's very possible he came from this household, and I would advise further research with the Library of Virginia on the family. They may have something in their records that indicates where the family came from.

Deloris