Its annoying as scout cause I have to rapidly move around to dodge other bullets while trying to take down the mini, and then the mini just homes in on me even if Im going at the speed of sound.
I'd say it's the most annoying as a scout or pyro. You can pick at it with a pistol as a scout, but you have low health, and turning a corner and getting hit by it can be pretty damaging, especially if you have your scattergun out cause it's horrible for damaging sentries.
As a pyro, your flare guns hardly damage it at all, so you have to be close. They're just annoying as hell, and get put up faster than you can take them down.