1. This looks like Patrick in 1870 with his family in case you do not haveit

Name: Patrick Ivey
Birth Year: abt 1820
Age in 1870: 50
Birthplace: Georgia
Home in 1870: Sardis, Bullock, Alabama
Race: Black
Gender: Male
Post Office: Union Springs
Household Members:
Patrick Ivey 50
Dilce Ivey 35
Augustus Ivey 12
Thadeus Ivey 10
Emeline Ivey 8
Amos Ivey 6
Michael Ivey 4
Noname Ivey 1

2. It seems they had a child named Evan Ivey who died in Union Springs, Bullock County, Alabama in 1931: http://pilot.familysearch.org/recordsearch/start.html#placeId=33&searchType=close&alivePlaceLoc1=Alabama%2C%20United%20States&surname=Ivey&givenName=patrick&p=recordResults

3. Bullock County was created 5 December 1866 from Barbour, Macon, Montgomery and Pike Counties.
Augusyus is listed on the 1870 US Census as being born in AL abt. 1858. Bullock did not exist then, so they could have been living in one of the other counties.

4. I would locate as many on that death index above and order the death certificates to see what is listed as the parent's birth place(s). I have been successful with that technique in the past when I wanted to learn more about the parents. Fortunately, you have lots of children to trace.

5. I would then go to this website to read about genealogical resources for African Americans in Alabama: https://wiki.familysearch.org/en/Alabama_Minorities

6. You will need to step back slowly in time and widen your search to include the whole family group, research any Ivey's with the same name in the area, esp white (Wills, and land).
Research records for both looking for a connection that leads to the original site. The 1870 Census says Patrick was born in GA. You will need to find other sources to prove or refute this.

I hope this helps.