People can DDOS which is distributed denial of service which is basically sending alot of packets to your router and blocks off your service due to overflow. Its almost like when you type in ping google.com in on your CMD but x10000. So yes, they can do something with your IP.