But what about people who don't want these approximations?
You don't run or join a server, full stop. That's the literal only way to prevent this. I'm sorry to be the bearer of bad news here but this information truly is in the public domain. If you access it you're merely making use of a service that others have provided and made extraordinarily easy and user-friendly to use. Any five year old can figure it out because if you google "location from IP" you get hundreds of easy-to-follow tutorials on the matter. If you have qualms with this the only way to stop it is to get your ISP to change your GeoIP location, and I don't even know if that's possible.
Also, I'd like to reiterate that until you actually say where you're located, nobody can ever know if your GeoIP location is off by 5 miles or 1000 miles. There's no way to tell how accurate it is until you
say how accurate it is, which would be violating your own privacy. It's why I haven't said how accurate mine is, for example (saying it's way far off makes it sound suspicious, too.)
You're basically signing an agreement form when you start a server that you're putting yours and others IPs out there
This is exactly what you're doing, yes. By hosting a server you are broadcasting your IP to the entire internet, out for anyone to see. By visiting a website you're broadcasting your IP to that website (and oh boy don't get me started on what websites do with GeoIP information), etc etc. And like I said, GeoIP isn't really useful if what you want to do is personally creep on someone because, as said, there's no way to tell how accurate it is until you say how accurate it is.