I don't really know. My Harvey's family is from Albemarle, VA. Lewis (1801) and Amanda Coleman Harvey (1815). We know Lewis worked At Monticello. There may have been a Jackson Harvey working there in 1867 also. However, there are no other family members in Albemarle names Harvey so it is difficult to figure out where Lewis came from. My guess is from Col. John Harvie, who was a in Albemarle Co for a while.
Any information regarding the family of your John Harvey would be appreciated.