It's because servers focus too hard on one idea that gets repetitive and boring after a while. My Falling Tiles server can retain 40-60 players all the time because I aimed for having a lot of content to go along with the gamemode, which encourages players to stick around between rounds and when they get bored.
Not only that, but I believe having a strong social aspect to a server also assists in keeping players around. Too many servers are full of people that are too focused on the game and don't care to talk to others. Add a lounge or another area on your servers where players can just chill and talk. Maybe add a couple arcade games or something in the lounge too.
It's not just about the quality of the main gamemode. It's about overall completeness and content.
This post was mostly aimed at people who actually put effort into their servers though. There are plenty of just plain stuffty servers that drive players away as well.