Yeah. It's a thing.
I don't put effort into looking attractive, but I keep a good handle on hygiene because that part is just good manners.
I don't shun people based on appearance, but it's often pretty clear when I'm the only person somebody is capable of holding a conversation with without being intimidated by the fact that they look normal.