I could never do a 1v4, let alone a 1v1 with anyone.
You say the lesser feat first, not the greater one.
You've just said "I can't even do this really really hard thing, so I definitely couldn't do the easier thing", which is dumb.
"I could never do a 1v1, let alone a 1v4".
This means you can't do a difficult thing, so therefore can't do an even more difficult thing. Which is a logical sentence and worth saying.