well, it depends on the age of the Charles in the census. If Samuel was born in 1861 and Charles was between 24-35 in 1830, that would mean Charles was 50-60 (give or take a bit) in 1861, it's not all together impossible for him to have been the father, but if they were even related, Charles likely would have been the grandfather.

My question though is what makes you think that these 2 men were related?