The only problem with your logic is that you are not directly affected by your great grandfather today. Blacks still have never gotten official compensation for anything their ancestors went through, and are still in the poorest areas of the country enduring racism and hardship today.
The fact of the matter is that the people who did inflict that suffering won't pay for it. I as a taxpayer will. I personally see no reason to pay for the sins of my forefathers, or rather, for the sins of other peoples' forefathers, mine fought for the North (worst thing any of my forbears did was shoot Ruskies for Riddler). The actions of those long dead men are out of my hands, and I see no valid reason to personally pay reparations for them as a taxpayer.
According to a Native American friend of mine, Riddler based some of his holocaust stuff around Andrew Jackson Trail of Tears business. I haven't done any research into it myself, but I'm just going to throw it out there.
Old George is still on all the quarters, even if the eagle isn't. I'd say that if Washington weren't already on the one, we should put him on the 20. Aside from him we should cut the duplicates (namely Lincoln by giving the penny the ax).
Madison deserves to be on the 20. During the War of 1812, he didn't strip citizens of their liberties like Adams did with the French problems; and even though the capitol was razed, the Brits still didn't retake the states, so he has a ~successful war under his belt.
Also, we should replace Hamilton with Cleveland, and Grant with Ike. Ike seems like an ok guy. Maybe replace FDR on the dime with Coolidge.