Why do the narrator and his family refer to his dad's childhood home as "the Middle of Nowhere'?

It is a remote place.

Explanation:

Although there is no precise reference to the text, "the middle of nowhere" is a phrase representing a very remote and usually isolated place. So, the narrator and his family must think that his dad's home is far away from them and possibly far away from civilization, hence they call it "the Middle of Nowhere".