The term "Egyptian" in colonial and early America had no reference whatsoever to Egypt. "Egyptian" in Europe and early America use is the derivation of the term "Gypsy". The ethnic origin of the Gypsy is India, not Egypt. In the Gypsy diaspora from India through Asia, through Anatolia, into Europe, these people were called "Little Egyptians". They arrive in 17th century America with the European invaders.
The Gypsy were indeed free people of color - misidentified in contemporary records as mulatto, white, black, Negro, etc. In 20th century literature they have been styled eroneusly as "Free African Americans".