There are fears that it could turn our society in to something like that of Orwell's 1984 (an Orwellian society). Plus, no organization is immune from hacking. If the NSA pools together even just their metadata, dangerous third parties could take that and use it against you. They could know where you get your daily bagel, for example, and also which kind, at what time, and what train you take to get there. From a simple database query, someone can know all about you and your routines. I believe a lot of concerns come from that; it's not just reading your emails, but connecting all the dots. If the NSA had their hands in just a single form of surveillance, that may not be such a big deal, but having tons of things monitored can be very dangerous.
I believe there were some cases of NSA employees using the metadata database for their own purposes too. Someone checked up on their wife and subsquently divorced her as they found that she was cheating, just from the metadata of her emails. Don't quote me on that though, it's been awhile since I've done my reading.
edit: Also a key part of how our government is set up is pitting power against power. It's largely viewed as state power vs federal power, but essentially the point that you yield your power is the point that you give them the right to oversee and belittle you. No matter how conservative or progressive you are, you should always be weary of sacrificing your personal rights and power.
