why don't you remember specifically when you drift into it?
Because you immediately go unconscious when you fall asleep. It's equivalent to being hit with a blunt object except without the concussion and long-term effects. Ask anyone who may have gotten hit by one and they'll probably say they don't remember getting hit or even feeling the pain.
How does you body go into sleep mode
Wall of text inbound.
When your body is relaxed enough, and you're in a suitable or exceptional environment dark enough and quiet enough, this gives the brain the signal that everything is a-ok and that there is probably nothing nearby to disrupt you whilst you're asleep. Around the time the brain recognizes this is usually when your brain knocks you unconscious to do stuff behind the scenes like memory assortment and other tasks to prepare you for when you wake up. After you fall asleep, around 20 minutes or less is when your body detects that it's not receiving any signals that it should move, it'll automatically paralyze itself (on some occasions this can happen while you are awake too, referred to as sleep paralysis), this prevents you from potentially moving around in your sleep that could say, cause sleep walking or damage yourself by trashing or falling out of your bed. After that your subconscious takes the responsibility of performing tasks throughout the brain and performing body tests (that don't require moving) to make sure everything is working (ie. repeatedly erecting the snake to make sure it's working properly).
And all that is just the process of getting into sleep, there is a lot more that happens after this one stage.