i don't think griefers will be a problem if you add a whitelist. only whitelist people you trust not to destroy everything you love and hold dear. you don't need to worry about me, i don't destroy things, i build MAGNIFICENT FOUR LEGGED SPIDERS OF JOY, and there's no plugin for that.
but seriously, you should just whitelist the server and add specific people. i think the main problem with the way the server works currently is that you just let anyone get builder for applying, and real griefers get thrown into the mix. if you keep it simple i'm sure a smaller population could keep itself from destroying the server. you could also do manual backups, such as making a manual backup every night, and if a griefer hits after dark, you just re-load the map to the latest nightly backup.