As far as balancing a musket goes, it could be balanced easily. It would be high ranged damage, but it needs to be reloaded after every shot, it takes awhile to reload, and the ammunition is expensive.
And if you want to make it a tad more realistic, you could have it where the muskets aren't very accurate, effectively balancing out the power with inefficiency. A well-timed shot could knock out a large chunk of health, but you would also be vulnerable after shooting it, and it would take a couple of seconds to get another shot off... And even then the next shot could just as easily miss.
It would be useful as a weapon to use when catching a player off guard, such as shooting him while he stops for whatever reason then jumping at him with a sword.
Of course, the bow already does a lot of damage; even more than a diamond sword! (A bow hits for nine hearts, fully charged. A diamond sword does seven.)