Just a further note: if the neighbor James Anthony is your gggrandfather's brother, which seems likely, it means that they may have been in MS as early as 1869, as he is showing as being born in MS; father VA (agrees with your gggrandfather, but mother born in MS (they may have had different mothers, or it is incorrect). There are 55 Anthonys in the 1870 AIS census index for MS.
What that means is that you should check for them in both MS and LA in the 1870 census.
The 1880 census indexes cannot always be relied on because they were created for a different purpose than the other indexes - persons are only listed in the 1880 if they had children of a certain age, because the government wanted to determine how many people might be eligible for social security when that index was created in the 1930s.
You may also want to check the SSDI (you can find that on either ancestry.com or rootsweb.com) for the children, especially the younger ones, to see if they are listed there. If so, you can request a copy of their SS5 (social security app) from the Social Security Administration. The price is steep though - $27, but it has a lot of useful information on it. Another cheaper alternative - find the names of the churches in the area in Marion County that they may have attended, and write them to see if they might have baptismal records for the children. It would probably be a good idea to offer to make a "donation" to the church for their time in looking up records.
I've had very spotty luck with churches - some are very cooperative, others not at all. Once again, best of luck.
P.S. I have the 1880 LDS CD set, so I'll check for him on that this afternoon when I get off work. If I find anything, I'll post it for you.