Alright so there is multiple factors here.
Above ground his castle is in fact bigger. (Although mine is increasing in size every day). My castle has taller towers though.
My castle has a much bigger and taller mountain than his.
My castle has more land mass and will have a city surrounding it.
My castle has a bigger underground. What my castle doesn't show is that underground I have a dungeon, torture chamber, throne room and I am going to eventually have some cataombs. The other castle has nothing underground.
Mine is a little more even out in every other category.