Gravity, I have an idea type thing.
If a weapon has a scope (XCR, RPG-7, L85, R700, etc), and you reloaded when you where sighted, it would go back to hip and then reload, but if it has standard iron sights, it would reload while you where aiming.
I just think having to go and un-sight is just way to time consuming and seems awkward.