Born on September 14, 1921, in New Haven, Connecticut, Baker was the ninth of twelve children whose parents emigrated from the Caribbean island of Nevis. While attending school, she was active in the New Haven Youth Council and the New Haven Adult Community Council. Baker joined the local chapter of the (NAACP) when she was denied admission to a local skating rink and public beach. With the help of local philanthropist Clarence Blakeslee, Baker attended Fisk University, a historically black college in Tennessee, before deciding to transfer to New York University. She graduated in 1943 with a degree in economics and went on to receive her law degree from Columbia Law School in 1946 and married Joel Wilson Motley, a real estate and insurance broker.