Food is the only limiting factor that keeps populations from growing too large

No, I don't think this is true,

Some other factor for example is female empowerment: more and more women choose not to have children or to have fewer children.

Also, housing could also be a factor: many families could decide against new children even if they can feed them, if their house is too small (think 40 Square meters and they already have 2 childen).