Let me get something out there:
To me, server management can be compared to owning a house. When guests come over, you try to be polite, and make them feel comfortable. If you open up with something rude and you're just trying to be joking with them, and it's all in good fun, make sure they know you are and don't get a bad first impression. Even though it's your server (or house, in my brown townogy), that doesn't mean that you can kick everyone out as you please without looking like a jackass, unless it's for a proper reason.
It's just something that I think.