You've got to deal with the following;
- Radiation (particularly following solar flares)
- Extreme heat on the moon
- Extreme cold on the moon/mars
- Having a steady source of food
- Having a steady source of Oxygen (growing plants for food will provide both of these)
- Venting oxygen safely (too much produced by plants greatly increases the chance of massive explosions and fires from even the slightest of sparks)
- Having a steady supply of water (water recyclers do a fantastic job, but eventually water will be entirely used up given enough time)
- Maintaining muscle mass in low-gravity
- Maintaining sanity/morale (particularly on Mars, where you are anywhere from 3 minutes to 22 minutes away at the speed of light, depending on distance between the two planets. This makes communication difficult, particularly any social interaction with people on earth. You have to get on with your fellow astronauts)
- Extreme dust storms on Mars
Plus much much more.