Are you sharing keys?It also might be that you are hosting at the same time, because when someone joins your server, you connect to the IP, and you guys probably have the same ports and IP's. (Most likely you have ports 28000.)
No we have different keys...
Find the prefs.cs or something and change the port in there to something different than your brothers.