What term did the muslims use to describe those who did not accept islam?

If they accepted Islamic rulership, but did not convert to the religion of Islam, they were called Ahl ad-Dhimma, meaning "Protected People". A subset of the Ahl ad-Dhimma were also called Ahl al-Kitaab, meaning "People of the Book" and usually used in reference to Jews and Christians.