To kill a mocking bird - Why does jem want to go back for his pants so badly?

Jem had to go back for his pants because the lie Dill told to Atticus didn't involve his pants being destroyed, only lost. He said he had lost them in "strip poker." Jem couldn't argue with that lie and come up with a better one where the pants were actually destroyed or else he would risk exposing the lie, so he had to go along with it.

If he hadn't come up with the pants relatively soon, Atticus would have punished him for losing them permanently, a punishment Jem seemed eager to avoid when he said he had not been "whipped" for a long time and he didn't want it to happen again. He clearly has a healthy respect for Atticus and is also afraid of the whip, as he should be. Atticus would have either punished him for losing the pants (something it would cost money to replace) or have punished him for lying, had he found out how the pants were really lost.

So, Jem really had no choice but to go back for his pants, as scary as that prospect was.