Yeah that would be bad; ~a few miles farther from the sun and we'd freeze, ~a few miles closer and we'd melt.
Earth's orbit is elliptical and the distance to sun varies between 147,098,290 km and 152,098,232 km. The exact variance is 4,999,942 km. A few miles here or there isn't going to do stuff.