A Good read
When Notch gets back into pace with his new company and eventually finishes SMP, with functioning PVP, mobs health etc. You and some companions building a small community will become the real-life equivalent of some Medieval travellers settling in a lawless frontier. If you know anything about colonies and frontiers in history you'll know that there were never any griefers. There were thieves, bandits, vandals and raiders. Soon this logic will apply to Minecraft. If a player walks into your small colony/town and begins attacking everyone and burning wooden houses down, he is not a griefer, he is a psychopath. If a group of people charge into your town on pigback and burn down your trees and crops, they aren't griefers, they are raiders. If a guy comes through a portal into your server and starts blowing things up, he is not griefing; he is committing an act of war. I could come up with countless scenarios, but an intelligent individual such as yourself will get the picture.
The point is that you have to stop trying to build a civilisation that will last forever, because it won't. We must accept that. On top of hostile players, creepers, spider and zombie raids will eventually wear down your town to the point where you will simply have to abandon it. Even if it does mean parting with your colossal golden diamond spire that took the entire server three weeks to complete. Take the Romans - they built huge monuments yet lost it all. Accept it for the beauty of the survival aspects of the game, civilisations will rise and fall. Nothing in the game will be permanent. Sure if you want to build a sprite inside an adminium wall that no one can see or get to with out getting out of the adminium cage they spawned in, there is always creative multiplayer. In fact, a simple way of stopping people from destroying your harmonious server is to simply build further away from spawn.
TL;DR: Soon the term "griefer" will be rendered obsolete, and rather than taking the easy route of kicking every player that is not friendly to you, I encourage you to find different ways of dealing with them - like killing them.