When doing genealogical research one works from known to the unknown. So continue to research James in the county and states records working your back - 1920-1870. Start by examining the county records (court, deed, probate, tax) of Montgomery County, AL.

Since you were unable to locate him in any census prior to the 1870, chances are he was probably enslaved. Therefore it is absolutely critical that you document the activities of James between 1865-1880. This is a significant time period for former enslaved individuals as they made the transition from enslaved to freed persons. In addition, many continued to interact with the slave holding families on various levels.

Also research James' neighbors as well especially the white Evans and Fraziers Families he is living near in the 1870 census. This will provide additional clues.

Some county records can be obtained through inter-library loan or through your locate family history center.

For a list of the county records that can be ordered from the family history center, please go to familysearch.org.