nobody really bats an eye if you're really into star trek, game of thrones, call of duty, indiana jones, etc. so i think the problem is more complex than "people shouldn't be openly so passionate about things they enjoy." i think the only reason for the negative response is this impression that some media is, in some way, not acceptable in a social context. like sports, books, popular movies, shows, and games, those are all pretty easy to get away with, but there's also this arbitrary classification of media that's considered outside the norm, and like with anything, people's responses to interests tend to be relative to what they think is within the realm of social acceptability. i used to have this kind of reaction toward Weeb stuff because i was conditioned to believe that it wasn't acceptable to be interested in that stuff, but now i'm past that cus it doesn't do me any good to sit and huff about people liking things, and it doesn't help people around me like me any better either.
