I just read a thread on rootsweb about this very subject. There someone states that the term was also used for people who were Jewish,Indian or any other non white.
I am trying to find info about a family on the 1820,1830 and 1840 census records in Jefferson county, Pa. The family is listed as free colored persons on those records. After that I can't find them any more, so what did they become W, M. B or N? I guess that is the $64,000 question.