The problem would be that the guards would have a horrid advantage over the attackers. The guards already have the "base" so that means they would have time to set up a defensive position. As attackers, you don't have the ability to find a well defendable position, nor would it be useful.
The proposed solution would be to do a timed deal.