The problem does not come from the game, but for the fact Magic Wand either selects too much (resulting in crappy edges) or too less (resulting in a white outline, or whatever the background of the player is).
Megashots take higher-resolution screenshots based on a scaling factor of up to 6x.
This means that if your scaling factor was set to 2, then any megashot you take (DOF or not) would be exactly twice the resolution you play on, and therefore twice as detailed.
Also, all you did to do your so called "edge smoothing" was use a Guassian Blur.
Dont use Guassian blur.
