Now I know you said you use these bricks to reduce the brick count, hear me out though.
Don't bricks with more polys like these lag more than normal bricks?
So you could have like 100,000 house bricks but it would probably lag more than 500,000 normal bricks