When a few bad eggs representing a group in any way mistreat somebody, the public is going to automatically assume this is how this group operates. Expecting the normal user to do more research on your group before labeling everyone in it a problem user is assuming that particular user is involved enough to do the research. The consensus on this skype call to the normal Joe is to avoid clique members because they're known to perpetuate drama. Why haven't you shown people otherwise? You can't just say things aren't that way, because nobody will believe you.
The issue on some members behalf is the lack of effort put into proving your group isn't full of problem users. I've said it before and I'll say it again, you can't change somebody's mind by telling them they're wrong. That doesn't make a difference. You have to show them why they're wrong. Disprove any allegations that your association with a skype call makes you a problem user by not acting how they expect you to act.
tl;dr version: People hate your call because they've been exposed to only the bad side of it. If you can figure out how to show them that there's more than just a bad side, you'll gain their respect.