The issues surrounding this are more complicated then you might think. Undoing supporting bricks could result in destruction that a build-trusted person would not otherwise be able to do. If you build a baseplate, then someone on your build trust list builds a castle on top of it, undoing the baseplate gives you the equivalent power of the wand.
