All that extra color data would have to be sent over the network. It takes 6 bits to represent 64 colors, but 32 bits to represent arbitrary RGB color.
Now here's where it gets interesting. Say you have 10,000 bricks in a scene, thats an extra 260,000 bits to go from 6 bit to 32 bit color. On the "DSL" network setting, the packet size is 350 bits. Thats 743 packets to send the extra color data. The packet rate for the "DSL" setting is 20/second. It would take 37 seconds to send all of that extra color data. And that's in an ideal scenario where you're maxing out each packet. In reality, it would probably take about a minute.
The bottom line is that 64 is more than enough unless you're trying to build some kind of goatse mural.