I believe it relates to the octotree, not rendering. Rendering a polygon 4x as large will only have a negligible effect on render time, and it wouldn't impact load times.
An octotree works by splitting a cubes in to smaller cubes essentially. By placing a 64x cube, you're intersecting much more of the octotree cubes (divisions? I'm only familiar with the concept, not terminology) than placing a 32x cube. The game has to do a lot of work to divide up the block. Imagine a trench war server, where you randomly split a x64 cube in to dozens of bricks by mining out a single hole in the middle.