Jungle is designed to be able to catch up easy. Even if you had a bad start, as long as you're farming and not wasting time with poorly-executed ganks then you should easily catch up with the rival jungler and still out farm the laners.
You've obviously never lost your first buff while jungling then. While it's possible to catch up, it's extremely difficult, considering A) you probably lost most of the XP from the buff, and B) You don't have the mana regen/cd reduction of the buff. As a jungler with high early mana costs like Fiddlesticks, Shaco, potentially Vi, etc, you'll be OOM and forced to go back every few camps, losing you a lot of jungle time.
The disadvantage isn't that it's impossible to catch back up, the problem is in the fact that you have to use your time to play catch-up instead of doing things to help your team, like warding, ganking, and split pushing/taking dragon.