I kept them clustered near the spawn by killing them when they did that (FYI Trifax and anyone else who's wondering, I let off occasionally in case they decided to stop. They just immediately went back to griefing). They kept punching everyone around them anyway, including me, and kept respawning, so a "militia" wouldn't be the most effective thing.