Still only on normal difficulty, but blimey, there's much more challenge if you skip all the early quests and head into the city while lower level.
It's definitely all the quests from Preston, sanctuary and other settlements that cause you to seemingly level up a lot by the time you get to diamond city.
I've had some really tough moments with super mutants and gunners/raiders in the city so far.