Neither do I, unless I shoot a hat and get a headshot. Sure does suck for those with the ghastly gibus or towering pile of hats.
I only use hats as identification.
For example, when there are two spies on the enemy team, one of them being Gamefandan. I want to know which forgeter will spinstab me any time I look at him.