In some ways, you are right, in other ways, wrong.
Sure, a couple of badmins might spark up, but they can always be punished. And if someone sees an admin screwing my server up, I added a de-admin form. I hope that doesn't get abused out of jealousy.
Can take that out later if it seems like a bad idea.
Besides, I put if they abuse the admin revoke form, the person gets banned for at least a week.