Please excuse me for add some thoughts on this. (I am a white jazz historian living in Norway!)
But this phrase is univeral and I have often been met by it by elder
family members when researching my own family tree.
But back to what I was thinking of:
Someone owned the ships that carried the slaves from West Africa and too many of these ship owners were Norwegian. A couple of years ago I was doing some research on this and began to gather information from the shipping companies. When the topic was how and when these big fortunes were made I was told: "Let sleeping dogs lie, those days it was legitime business."