We were carrying the tools and things like swords and the crafted goods. Fish just had the raw building supplies because he was the one digging out massive holes while we were getting other goods. If you're going to ban him at least transfer his inventory to a chest or something. If we don't have those supplies we're going to have to deforest the whole area to build our structures.