You make an interesting point. I had some relatives who aged 20 years from one census to another even though only 10 years had past or they aged 6 years, or in one case the person claimed that they were the same age as they were 10 years ago. Then I had to start thinking like someone who would have lived during the time period of 1870 through 1930, or at least I tried to think like them. Memories have a way of fading or clouding as we age. When I was 10, I could name everyone in my first grade class, now I barely remember the teacher's name. I imagine that it might have been the same way for some of our newly emancipated ancestors. Even their owners, those who kept good records, might not know how old they were at the end of the civil war. I think, for the sake of argument that it is probably safer to go with the oldest recorded age.
Here is another more recent example; my grandmother is 94. She did not have a birth certificate. In order to prove her birth, she had to get a copy of the 1920 census with a raised seal on it. She shows this in lieu of a birth certificate. There is no family bible and my grandmother's mother died when she was 4. I think she told me that she "picked" out her birthday when she went to school. We are pretty certain that she might have the correct month as all of the other children also claimed to have been born in the spring but who knows? We celebrate the day she picked as her birthday. After 94 years, who's going to argue with her? LOL
Just my thought.
Peace and Blessings.