Why was the purchase of the Louisiana territory controversial to many Americans

"The controversy didn't end there. People were already living in the Louisiana territory, most of them French, Spanish, or free Africans ... Since the Louisiana Purchase was part of a treaty with Napoleon that Jefferson was entering into, it could not be unconstitutional."