That's true KOL, but not how the today's society thinks.
Actually visual grouping is how nature operates. If you see some shady-ass guy with an overcoat in an alleyway, looking around suspiciously, you avoid him. He could potentially be the nicest, most generous guy on the planet, but then again, visually he's a coke dealer.
This is also true in nature, where bright colors mean "do not eat me I am poisonous you dumb motherforgeter", except on butterflies.
No, you cannot brown townyze someone's character visually, but you can guess, and then supplement that hypothesis with observation.