Your mentioning that one of the stories about your family was that they were of Welsh origins. It's strange but, that may actually have something to do with the origins of some families later considered as "Mulatto", Native American "Indians", or even African Americans, judging from some of the things I've found in my research. One family line in particular, EVANS, were first found in Virginia, then many moved on to North Carolina, and all of the ones I've researched were always among the "Free people of Colour" as far back as can be found in the 1700's. But the very first person in one particular line, Morris EVANS (b.1675-d. 1739 York Co.,VA)was first listed as a Welshman, but in a 1738 court case both he and his (wife?)were brought up on charges for not declaring both of them as tithables. Now, for the wife to be a tithable at that time, the law was that "wives of free negroes, mulattos, and Indians" were tithable. (By 1748 the VA law was adjusted to, "all males of 16 and upwards, all negroes, mulatto, and Indian women aged 16 and upwards,as well as all wives of free negroes, mulattoes, and Indians")
So, it does appear that there may have been something to the idea that some Welshmen were either of such appearance to have been mistaken for mulatto, or that some may have intermarried with Native or African American women during the early Colonial days.