One of my old friends was presumably picked on for being fat so he took karate or some stuff. He then became a really big starfish/psychopath and would literally walk up to me, yell "BITCH SLAP," slap me as hard as he could, then laugh hysterically. We are not friends anymore.
The same kind of goes for the "friend" that would come up behind me and strangle me for fun, but he's a bit of a different case.
On topic: I can definitely see why you don't want to be his friend any more.