My Layfield ancestor came to West Virginia from Maryland in the very early 1800's. He was white and the family in WV did not own slaves. He was born in the 1770's in Maryland, according to what he told the census taker in 1850. The Layfield family came over from England in the late 1600's and settled in eastern Maryland on what is called the Eastern Shore also called the DELMARVA Peninsula.
The Layfields in Maryland were slave owners, so you might want to search there. When slaves were freed, some took the name of their former owner. But the family and their former slaves have had a long time to branch out. Of course it is possible the black Layfields are Layfields in blood as well as name.
Good luck in your search.