Decided to play and see, informally, what ancestry indexed for the 1870 Census.

Ancestry's indices for the 1870 Census are sortable by white/colored. In Minnesota, at least, Indians were not counted as colored.

In some of the other states, Chinese made up a large proportion of those indexed as "colored" (number indicated is those listing birthplace as China)

These are pulled from a quick search at ancestry (has a search choice of white or colored) of the 1870 census:

State # of Colored Residents Total

Arizona Territory 21 9658
California 2846 559,135 (a curiosity...there were 48,012 Chinese, but only 3 were included in the "colored")
Idaho Territory 44 15,001
Kansas 11,845 364,677
Minnesota 624 587,707
Montana Territory 1,815 20,617
New Mexico Territory 175 91,756
Oregon 3,334 (3,116 Chinese) 91,071 Texas 220,435 820,700
Utah Territory 543(407 born in China) 86,787
Washington Territory 434 (226 born in China) 23,966
Wyoming Territory 178 (122 born in China) 9,129