I sleep with my street close all the time.
Belt, jeans, white T-shirt... Pretty much everynight. Sometimes I take my belt off.
I have woken up with my button undone and I know its no one because my rooms a minefield of a mess so I would hear someone coming in and my window is to high for anyone to climb in and out. Not to mention its blocked.
Your just paranoid chillax bro.