it's hard to explain but in photoshop i make a square in the specific colour, rasterize the square for the background, trim the image i'm putting in it, drag a selection box around it, make the selection non-trans pixels only, and trim the background with the selection from the other layer
what I'd do is:
set flood select tolerance to 0%
change color of background with fill can
select background with magic wand
invert selection
delete selection
add some FXAA with an effect