Graphics are more of a distraction.
Not necessarily.
Every element of the game, be it mechanics, aesthetics, technology or narrative, is crucial to the success of the final game. In the perfect world, they would all exist equally and in harmony, and many professional studios with lots of people, experience and resources do make them work well.
When you're just a one man army, you have to play the game of development smart, and focus on things that will make the biggest impact in your development cycle first. Of course, it's important to acknowledge what kind of art you'd like and how you could integrate that into the mechanics, but at the same time, if you're planning on keeping them as separated elements, than you need to prioritise what's going to hold your player in the world the longest.
"The name of the game is the
game."