In the perspective of fairness, it shouldn't be removed. People don't like stuff they don't like, but we haven't banned politics or religion because many people hate Republicans or Christians. It exists, intense discussions and arguments take place and no one ever wins.
From a perspective of avoiding problems and attempting to make everyone happy, removing it still wouldn't work. You'd only make those who hate it happy.
Personally, if there was a way to organize a "move" that the others would accept, it wouldn't kill me to do it. This is a very unlikely situation though, most if not all of them enjoy this forum and prefer to keep all the comforts of home in one place. Even the idea of creating an invisible sub-forum probably wouldn't fly.
If I could, I'd attempt to figure out a solution that both sides would be happy with. Unfortunately, that's neither my place nor am I sure I'd be able to do that.