look it may be his server, but the reasoning of "his server, his choice" doesn't justify it. if you murdered someone in cold blood and told the police, "My home, my rules" do you think the police would say "you kmow what? your right, My bad" no questions asked? NO
That depends on two things:
Is it his property? In the case of this being his server, then it is.
Is his being threatened by the OP and his friend? No. Even from a legal parallel it doesn't work.
In the case of online game hosting, you are not obligated to play on the server, and frankly are not entitled to the presumed right to have a good time. The host could put you through hell in-game while keeping his actions everything short of illegal and have nothing wrong with him, with the exception of breaking moral codes and community-defined societal standards.