The old GPU is that card-like thing at the bottom of the image, underneath the fan attached to the motherboard via a slot.
In order to remove it, you'll need to push on the tab like thing at the end of the slot. You'll also need to check for any screws that are keeping the metal plate on the side of the GPU attached to the external housing. Aside from that, you can simply pull it out.
Put your new GPU in the same slot doing the reverse of what you did to get the old one out, and then attach the appropriate cords as seen in the manual.