Why do people object to banning discrimination based on appearance?

People object to banning discrimination based on appearance because of people who are diseased, maimed, mutilated, or deformed where exposing themselves in public. It is often seen as objective and universal, like it is upon the person's decision to whether look at somebody based on their personal judgment.