the best solution for this would probably be to do a single boxsearch that encompasses the entire volume of the cone, then do some math on any objects it picks up to test whether or not they are actually in the cone. this would be difficult mathwise, but its probably the most efficient and the most accurate way to do it.
im not too sure how to test if a certain point is inside a cone or not, but one way would be to find out how far away from the muzzle the object is according to the center line of the cone (this most likely requires trig), then use that distance to calculate the radius of the cone at that distance using a cone equation (im sure you can find one on google or wikipedia), then see if the object is in that radius, probably using vectordist(pos1,pos2).