if the user doesn't know what the password is, the user shouldn't be inviting other users
Ok ... To make this more simple for you, imagine the following scenario:
1. User joins a server with a password.
2. User invites a friend.
3. Server admin changes the password.
4. Friend tries to join.
That's a perfectly legitimate scenario which just cannot work with what you're proposing.