I use Add-Ons since I usually can't replicate their effect in a good enough manner without them.
I do however, always keep my Add-Ons down to the bare essentials.
I will almost always use JVS, Round Corner Bricks and Pole Bricks since I find they add a good effect and are exceedingly handy in certain cases of building.
Otherwise, I keep weapons on that I need, which are always ones that fit my specific genre of my server. If I'm doing a Medieval thing, I'll have swords and bows and things. If I'm doing a Western, I'll do Bows and WildWest guns. So on and so forth.
Other than weapons and bricks, my Add-Ons will generally remain at Tools. Duplicator, Fillcan, /Pickcolor command, etc...
The basic things that help speed up building.
The rarest Add-On for me to have though, is probably emitters and music/sounds. I've used emitters a couple of times when I don't have anything default that will work, but they're only for minor things. A Candle Emitter or something else that just doesn't look right with any other default emitter.
Music, I've only ever used once, which was just a test of a piece that I had ripped and packaged myself. It was ok, but not really crucial.
Anyway, I keep my Add-Ons to the minimum in order to ensure that people don't spend ages downloading stuff (Although people still complain when they have to download three Add-Ons) as well as to ensure that my server looks nice and things run smoothly.
If people can't be bothered to wait for a few Add-Ons to load before they get to join in on the server, then they're not welcome. I don't really want to play alongside someone who's so impatient that they can't wait a minute or so.
I also have strict no spaming rules for things like Items, sounds, music and emitters. Generally this is done by the fact that if I'm doing a roleplay, I give people Hammers and Printers, but no wrenches. If someone needs a Wrench, they ask and it's given to them if they have a building that is deemed worthy of having a wrench and if it's completed to an acceptable degree.
Basically, I get by due to the nature of the things I do on my server, which will allow for me to monitor and control what is used by people. I obviously would have more difficulty if all I ever did was Freebuilds or similar such things.