Are you using a fairly old computer? I've noticed when the first person joined my game's after I just started hosting it would freeze for around 20 seconds
Happens to me too.