I checked the 1850 census you mentioned and that Isham Smith was a White male age 63. Generally if the box was blank the person was White. I checked back a few pages and found some people who were listed with a B or M in the box alongside others with blank boxes and there was also a notation at the bottom of the page that said "3 col'd males" next to the totals for (white) males & females.

Considering that Isham is 63 in 1850 there is a chance that he died before emancipation and the slaves may be listed in his estate inventory. Hope this helps!