Other recommendations:
John Hope Franklin's "The Free Negro in North Carolina, 1790-1860", 1943, reprint 1995
Larry Koger, "Black Slaveowners: Free Black Slave Masters in South Carolina, 1790-1860", 1985, paperback 1995
"The Known World" is a novel about such a family in a fictional county of VA, by a writer in Washington, DC (can't remember his name right now.)