Improve the walls by putting more method and structure into them. Look at other builds and how they arrange their walls. Maybe do a brick pattern or some support beaming, just be sure it looks nice and is intentional.
You're allowed to use lots of bricks, so long as it has the intention of looking nice and isn't... Uh... Ugly?