There were certainly several slaveholders in Accomack surnamed Selby - if you do a google search using keywords "Accomac Selby negro" lots of interesting things pop up, like the following:

I Give and devise to Colo. WILLIAM SELBY the following negroes, viz: GUY, _LEY, JAMES, LITHOR and her Children, which I exchanged with GEORGE HOLDEN during our lives for a negro Woman called FLORA and her children, and if LITHOR and her children are so disposed of as not to be Conveniently procured then I give FLORA and her children to the Colonel, also a Case of Drawers, a large Copper Still and a set of Curtains.

Check the Accomack County Register of Free Negroes, 1785-1863, unpaged, Library of Virginia microfilm reel 316:no.248

See link below for more records