Yeah, residue isn't really a problem unless you handle parts immediately after eating greasy food or something, lol.
Grounding, however, is always important.  I ground myself like every other second when I'm handling parts.  Not when I'm holding them, though.