Yeah I was thinking about that. Plus, a 454 is a big block engine which is focused it can take the more on torque rather than horsepower. Still has a good bit more HP nonetheless.
slap a turbocharger on that thing, get bigger fuel jets and remove the choke along with new heads, pistons, heavier cam, bigger intake, open headers and anything else you need to make sure it can take the turbo and you'll be in pretty good position. No need for an engine swap.
Edit: actually thinking about it you really could run off and leave a built 350 with a stock 454 but it really comes down to the money. 454s are a lot more uncommon and expensive to upgrade but they are definitely some hosses.