Rotating counterclockwise about the +z axis (i.e. left/right):
%x = getWord(%muzzleVector, 0);
%y = getWord(%muzzleVector, 1);
%z = getWord(%muzzleVector, 2);
%a = %x*mCos(%angle)-%y*mSin(%angle);
%b = %x*mSin(%angle)+%y*mCos(%angle);
%c = %z;
%vec = %a SPC %b SPC %c;
This only works if your player's up vector is the +z axis (so it won't necessarily work correctly if they're in a vehicle). You can adapt the below code if it's important that it works in a vehicle.
Rotating around the player's right vector (i.e. up/down) is a bit more fun (your definition of fun may vary) and looks like this:
Where the player's muzzle vector is (x,y,z) and the player's right vector is (u,v,w). The angle of rotation, of course, is θ. Since the player's right vector will always be a unit vector, u
2+v
2+w
2 is always 1, simplifying things a bit. It would be possible to simplify it a bit further if you assume that the player's up vector is always the +z axis, which would make the right vector always lie in the xy plane. This assumption, however, would make the operation not necessarily work if the player is in a vehicle.
This monstrosity should do it:
%rightVector = vectorCross(%forwardVector, %upVector);
%x = getWord(%muzzleVector, 0);
%y = getWord(%muzzleVector, 1);
%z = getWord(%muzzleVector, 2);
%u = getWord(%rightVector, 0);
%v = getWord(%rightVector, 1);
%w = getWord(%rightVector, 2);
%a = %u*(%u*%x+%v*%y+%w*%z)*(1-mCos(%angle))+%x*mCos(%angle)+(%v*%z-%w*%y)*mSin(%angle);
%b = %u*(%u*%x+%v*%y+%w*%z)*(1-mCos(%angle))+%y*mCos(%angle)+(%w*%x-%u*%z)*mSin(%angle);
%c = %u*(%u*%x+%v*%y+%w*%z)*(1-mCos(%angle))+%z*mCos(%angle)+(%u*%y-%v*%x)*mSin(%angle);
%vec = %a SPC %b SPC %c;
(Source for second part)EDIT: To generalize this, replace %muzzleVector with the vector you want to be rotated and %rightVector with the vector you want to rotate your other vector around. The vector you're rotating around needs to be normalized (i.e. length 1) or weird things are probably going to happen.
Note that mCos and mSin both require the supplied angle to be given in radians.