I think I would, but imagine how bad everything would be if there was ever a tornado, mass flooding, a blizzard, or any type of natural disaster. I doubt that even 3 layers of LEGO bricks could hold something together.
A photograph of a home made out of LEGO's:

A video of the LEGO house:
http://www.youtube.com/watch?v=OC1nTGje0pwJames May, the Top Gear presenter and the host of his show Toy Stories constructed that house from pure LEGO's. Yes, I realize it does look quite silly, but it's fascinating how a house could be built from just a basis of LEGO toy bricks, it's remarkable.
It would probably be very unrealistic to have a whole city made out of 100% LEGO bricks, but who knows what could happen in the future.