There are a couple things that generally turn me away from a server immediately.
1. Previous experience with the server. I don't join servers I know I don't enjoy it.
2. Massive load times. I turn off all my downloads. If I have to sit through 2000 files to get in, it's probably not worth it, and there's a large chance something is going to go wrong and i'll get a connection error. Manage your add-ons properly and disable add-ons that are not crucial to the server.
3. Once I get in, I take a look at what's there. If the server is really empty with little to no bricks, I generally leave immediately.
4. If there's something for me to do as soon as I get in, the chances i'll stay skyrockets. This is important, because there are MANY gamemodes, especially prison escapes and things like Despair Syndrome, where you join mid game or early game and have to wait 5-10 minutes flying around before you get to do anything. It's boring, and a major turn-away. Avoid that stuff if you can.
EDIT: I joined Nightbriger's apocalyptic RP earlier. It made me walk through a bunch of doors and fed me information in the form of default red centerprint text. Took several minutes. Don't do that.
5. Ping. If the ping is above the 250ms range, it's really difficult to play on the server.
Beyond that, it boils down to the details. Is it clear what's going on? Is the build an eye-sore? Is the gamemode fun? Are people socializing?