Jews and people of color are considered to be a minority because :

They have been looked down upon for decades and possibly centuries. It had become a sort of a social norm in a sense to view them that way, hence all of the public opinion has shaped how we think of them today.