Somewhat abusable, people would purposely rate down servers being jackasses. Though comments on a server may work, but it'd could only be shown to the players and not to the host.
I think it would do more harm then good, however I wouldn't mind seeing a couple of vote buttons that users could only vote once for, for example "This server is" Great!, Bad, Boring, Good Building. But players can change there vote when ever, but they can only have 1 vote. So in the rating list you may see 10 votes on Good Building but 3 on Great!, sort of stops idiots spamming votes on Bad. 1 Vote per ID that's changeable.