Ok those walls. They weren't really meant for defense, they just surround the courtyard and act as a fence so you don't fall off the side of the island. I fear if I change them it will make it seem too enclosed at that point and I will need to add more in the courtyard to fill up some space.
If a castles walls cannot defend then they are useless walls that shouldn't be there.
Traditionally castles would have 3 or more walls each separated by a deep trench filled with nasty things like spears and spikes and plagued rats to add to the defense, if it wall is so short that people can jump over it the what's the point? It wouldn't even be able to fend off bombardment by artillery such as ballistas and catapults.
I appreciate that this is not real life, but a castle just isn't a castle without enormous walls to keep out invaders, it's more of a holiday home