Also, I do have the right to ban anyone I want for however long I want, because its my server.
/thread
It's not the fact that you ban people as you did, but that you did it suddenly without telling them or putting a valid reason in the ban message.
People aren't annoyed about being banned, but being banned in a way where it seems you have some unknown and unwarranted grudge against them which they don't understand and you've neither explained nor even openly expressed it. That's not nice and people don't like it.
Ban people all you like. But don't be surprised when they're angered and upset that you do it without warning or giving reason. Even putting "I don't like you" as a ban reason is better than a vague phrase.