Was he blocking your exit?
Was he following you at any point?
What was the tone in which he asked?
It sounds to me that you're a little too sensitive to this. Whether or not the question seemed dubious, your explanation doesn't really merit an incriminating action.