I believe it gives you the right to set the rules to anything you wish, and from there you could permaban anyone you wish. Assuming the rules are properly posted, I would be fine with the owner having a rule such as follows:
"I reserve the right to ban anyone for any reason."
I would just leave, but it's acceptable ethically.
Therefore my answer is "no", as it does not give you the right to randomly ban anyone.
However, due to ownership, you DO have the right to do whatever the forget you want on your server.
This doesn't mean you SHOULD do whatever the forget you want with no regard to your guests, but you CAN.