But if I removed "only" then that would make it a completely different meaning.
Calling people "Only children" could mean two things: That they're just kids or are the only child in a family, whereas simply calling people children is being condescending.