How about make them hostile if you take wheat or break the buildings.
no, it would be cooler if you either had to stay away from villages completely or raid them and kill the inhabitants. imagine they were like vampires or some stuff, it would add a scary aspect to the game because you know there are other humanoids out there capable of kicking your ass. also, imagine just taking over the village, taking their wheat and supplies, burning houses, and killing the inhabitance with a group of friends as they fight back against you. it would be amazing.