Although I shouldn't be annoyed by it at all, as I know that some people join servers by accident and therefore leave, but when people join and immediately leave without even saying Hello, or Sorry wrong server.
I always feel rude and alien if I join a server, don't say anything and then leave.
I usually do this because I'm, probably, a little shy or just don't know what to say, which usually only happens when i go into a server with a lot of people, who are already in discussion.
I also hate going into a server by accident, and it's always a newer persons server, who has no one there, and then having to say sorry and admit I made a mistake before leaving.
I have to leave, since I'm supposed to be somewhere else, usually a friends server. But, I don't like getting their hopes up (And sometimes you defintely see that they've gotten their hopes up) and then crushing them and leaving.
Sometimes I even stay, even though it's a terrible server, just because I don't want to seem so rude to join someones server, who has no one else joining and then leave.
This comes from being a bit depressed when I host, and usually I don't get much traffic unless I invite some people.