So is it only when you are in a minigame, or just playing in general. The way you phrased it makes it seem like it only does this in minigames.
I've done dwelled much into this, but have you checke out your keybinds for turn left/right/up/down? Also, did any weird error/glitch occur involving extreme lag happen before your issue? I had a similar issue with turning down after an extreme lag after duping happened.