Don't get me wrong, I don't hate everybody who watches MLP, except for the people who blatantly show it off using avatars and pictures that are outside of the MLP topic. I want to be able to browse the internet without having to see something that I'd rather not see.
It's annoying.
Funny you say that, because in the first paragraph of your post you're stating that, "The reason I hate MLP fandom is that it distracts me from more interesting portions of the internet". The minority of people who watch the show, and the fact MOST of it is contained in one large thread results in you almost never having to see/read any of it. Unless you frequently browse /b/ or ponychan, you won't have to worry.
Rather, the mentality resulting in your hate for people who use avatars with ponies is just because they enjoy a television show you can't enjoy, so you subconsciously render it bad to show any form of appreciation for the cartoon.
In which case, stop it because that's rude. So is staring. Don't stare at strangers.