Setting it by ping as Epicduke said is the best way to combat this.
But bear in mind that you'd have all "---" servers at the bottom of the list, where you might not look.
And it might just be that there is a good server down there with people on it.
Best bet is to look at the Ping and the amount of players on it.
Also, out of curiosity, Magnusbon, how would you redesign it, if you could?