Can you not just ban anyone who has visited your server? I feel like there's a console command to fetch the IP of past connected clients with their BLID along that, and you could ban them on that.
stuffty behavior sure, it's unfortunate that preventing this would require technical skills that most people don't have.