Alright, increased map size to 1024x1024.
Thing is, no matter how big I make it, nothing slows down except initial download time, so I might make it even bigger.
However, that initial download time is now about two minutes, and that's when the server and client are on the same computer, no lag.
So I've tried porting the map downloading to a binary protocol, but it doesn't really mix well with the text based protocol of everything else.