You must factor in that former slaves may have used one surname for the 1870 census, but a different one for the 1880 census.

Also, back then, with most of the recently freed slaves, illiteracy ran high, and exact birthdates or even birthplaces were not known. Sometimes I think they may have just made up their info, not giving much importance to the purpose of the census.

Trust your instincts, visit and revisit your notes and documentation, and don't forget the neighbors. Many times I have found elusive ancestors with names butchered in each census, but closer scrutiny reveals the bad handscript to be my research targets based on their same neighbors.