Its the person joining receiving all your add-ons decreasing your amount of add-ons can decrease freeze time
I doubt thats the issue, but theres a lot of possibilities. We need more specifics:
For one, is your computer good?
Have you been able to create servers that others have been able to join?
If you have, what were the most recent add-ons you've downloaded?
Etc.