you dont need to rush houses.
i start off by just placing the building zones on the ground and not even having walls.
my new game here i have 17 guys 4 heros, on the second dungeon level, and i still havnt made walls.
you dont even get a happiness bonus unless you have a ceiling. walls arent important. but even then its just a tiny bonus, build a few decorations in the room zones and let your guys idle (not have a que) and the happiness will go up fast enough to spawn more guys into the town.