Hi, I don't have access to the Rappahannock document but I did do a search for your John Lillard.
First I checked the slave schedule for 1860 and found a slaveowner by the name of John Lillard living in Madison county Virginia. He owned 1 slave male, black, 9 yrs old.
Then I checked the free inhabitant census for Virginia 1860. I didn't come up with Rappahannock county. This John Lillard lived in Madison county, Criglersville. He lived alone, no wife or child listed. His occupation was listed as clerk.
I tried to find some maps of that time and it would seem that the 2 counties are close to each other or maybe next to each other.
Also, there was 1 white family of Lillard in Rappahannock, Washington, Va. The father was James, son John along with other family members.
The 1860 slave schedule suggests there was no Lillard slaveowner in Rappahannock, only Madison. It is possible that the county line changed or shifted over the years.
By 1870 there were many Lillards black/white/mulatto living in Rappahannock.
The 1850 slave schedule suggests all slaves owned by Lillards lived in Tennessee. Again, check boundary lines.
Hope this helps.