Ok so I had no idea my modem even had an access panel, so this is progress.
And better yet none of my forwarded ports are in there so I'm sure that's the issue.
edit: So it turns out the modem can actually be a router too and was set to modem+router mode, but if I switch it to be just a modem I lose connection. Ideas?