Why are you friends with people who send you useless group invites?
I used to play terraria multiplayer alot and I would make threads on the steam forums. I would get tons of invites and I accepted all of them.
The stray ones in my friends list are still there, They do that stuff.