Aeris, the fact is that it doesn't really matter if you don't care what anyone thinks or not. That really doesn't stop them from thinking it in the first place, now does it? You can deny it all you want, but everyone that has knowledge of it is judging your previous actions, and such actions have left a big enough impact that to this day people hold it against you.
You don't just 'drop' stuff like this and get away scot-free. You did something really, really stupid and because you did, people are going to judge you for it. It's never going to go away, and probing the issue is only going to make it worse. By going out and admitting to be male, you've developed everyone's negative view of you.
Do yourself, and everyone else a favor and just leave again.
