You can't say that backwards time travel isn't possible just because of cause and effect.
Cause and effect and merely inventions on the human mind, the way we perceive time. I can't say that I know it for sure, because our universe is far to vast and complex to say that something doesn't exist just because we haven't seen it (God withstanding, or rather, the Christian God doesn't exist, I know that for sure). It's very possible that there are beings that live in the fourth dimension and die before they are born. It's quite possible beings like Dr. Manhattan exist and can perceive time in a way that he can see past the events of the present. Hell, there's a possibility that out there is Galifrey and the Doctor and his TARDIS are traveling as we speak. We don't know what's out there, and we can never claim we do.
Physics are a set principle, but then, physics can be changed and broken. For instance, at one point in time we used to think that the atom was indivisible and the smallest possible thing in our universe. A hundred years ago if you argued it could be split, people like you would pop and explain how it wouldn't be possible. And then we split it and destroyed half of Japan. Twice.
Science evolves. That's what makes it such a wonderful field to use as understanding things. Saying something isn't possible because you know in your heart it isn't is like saying God exists because the Bible tells you so. Cause and effect can still exist, by the way, just not in a sense we see it as. We see it as cause leading to the effect, but then what if you just did it in reverse? What if you caused something to happen before it happens? You're still causing something and the effect is still there, just the time, or at least what we conceive as time, wouldn't match up.
Edit: Time is like a big ball of wibbly wobbly, timey wimey stuff.
