when far away though, wouldn't doing this cause z-fighting issues
Yes, especially depending on how close the top layer of faces are to the bottom layer
The closer they're together, the sooner the z-fighting happens, but the further away they are from each other, the more noticeable it is that your bricks have floating geometry that may noticeably clip into objects that are on your bricks
So there are potential issues either way, unfortunately