The jitter from the motion tracking is enough to give me a headache. I did this myself a while ago, and one thing I learned is use as many reference points as you have time to set up.
Seriously, cover the entire loving page in black dots, and use as many sheets of paper as you can. The more the merrier (unless one of the track points misbehave, in which case just stop tracking it altogether). Pretty sure that using different colors also helps, as you can set up filters which I'd expect to give much more effective results.
