For "raw" numbers on Rowan Co., NC in the Census as reflected on Ancestry here are the figures for those enumerated as non-white from 1850 to 1930.

Results
116 1850 United States Federal Census
138 1860 United States Federal Census
5,279 1870 United States Federal Census
6,368 1880 United States Federal Census
8,259 1900 United States Federal Census
9,107 1910 United States Federal Census
9,271 1920 United States Federal Census
3,837 1930 United States Federal Census

For statisical breakouts on the ethnic populations in various census years by State, County & Township try:
http://fisher.lib.virginia.edu/collections/stats/histcensus/

Not sure if these sites/figures will get you to where you want to go in this type of research but, hope they are of some help.