I don't know if you've viewed this book yet, but you should look at William E. Bigglestone's, They Stopped in Oberlin for info on John E. Patterson's family and migration to Ohio. Bigglestone dates the migration as taking place in 1856, which fits with Patterson's census appearances in Fayetteville and Oberlin. Unfortunately the book doesn't site sources, and I haven't reviewed it in awhile, so I can't recall where Bigglestone obtained his data.

Where did you find info on your ancestor's origins? Was it oral history or was it from published sources?