Yeah, residue isn't really a problem unless you handle parts immediately after eating greasy food or something, lol.
Grounding, however, is always important. I ground myself like every other second when I'm handling parts. Not when I'm holding them, though.