What arguments did opponents of annexing foreign territories present

The two main arguments that the opponents of annexing foreign territories presented were that 1) it would displace large groups of Natives who had nowhere to go, and 2) that it would lead to an increased debate over the imperialist nature of the US.