I think about it this way. If a fire starts in your house and you wake up and have to get out, do you want to stand outside naked in front of all your neighbors who will most likely see the fire and have you bang their door for a phone.
That is why I wear boxers and a wife-beater shirt.