Lots of people do. You just don't really realize because unless you're friends with them or something they just kind of slip away unnoticed. Then when you finally do realize, they're gone for good.
I've lost a lot of people I liked that way. Sometimes they come back for a short period of time (e.g. Miga, Hugums) and other times they just straight disappear.