Shift+Delete on an item to permanently delete it (skip recycle bin).
They removed this one on Windows 8, it made me very sad. But they also got rid of the confirmation dialog when you hit the delete key to recycle something.
Actually, I think it's just an 8.1 thing. Maybe it'll be back at the full release.