Anyone who does not build their home out of a stone of some sort is just asking for it to be burned down.
Also I think XP should be used in a skill system like Fallout 3 (maybe the others too, but I've never palyed any of them and only seen Eksi's LPs) where there are different skills, i.e.
Mining: The higher the bar is, the faster you mine. (when it is full, a diamond pick will mine faster than a golden, iron faster than diamond, stone faster than diamond and wood as fast as stone currently is)
Hunting: The higher the bar is, the damage you attack passive mobs with is boosted. (when it is full, fists do 2.5 hearts of damage to them)
Swiming: The higher the bar is, the faster you can swim. (when it is full, you can swim like a pro [about 4 blocks a second)
Agility: The higher the bar is, the faster you can run and jump higher. (when it is full you can run twice the normal speed and jump 2.5 blocks high [2 blocks longer when sprint-jumping])
And more I'm too lazy to list, and to balance this out XP orbs are rarer (passive mobs do not drop any, hostile mobs drop 1 and ender pearls can be crafted into 2), and dying would make you lose your XP orbs, along with any ender pearls you had.
It might be a little too nerfed but I don't want it to be easy because of the rather OP skills (which really can't be nerfed otherwise there is no point in getting XP orbs)