No one cares or even notices them in any argument.
They're just tools used to nit-pick every argument to make sure no point is "valid."
But they're simply tools to help us learn moral and logical reasons for doing things, not doing things, having certain view points, etc.
However, you are
partially right, sometimes people assume if there is a fallacy, the entire argument goes to stuff. (see link)
For example, see
this picture.
A fallacy in this story is the
argument from ignorance, which is assuming a claim is true because it cannot be proven false. (or vice versa) It's a fallacy because at that current time, or in any time, something may not be provable or unprovable, and thusly we must progress until we can prove/unprove the claim.
