I don't think it's necessarily "crazy", but it's definitely an idea loads of people dream and talk about but then never actually make a goal out of it - for whatever reason - especially when you're talking about living in the places you travel as opposed to visiting.
Personally, I have had such a goal for around two years now; I've got a couple thousand saved so far for travelling, and have also done lots of research. I know I'd be good off in many cases, because as an IT nerd/designer I can pretty much find good-paying work wherever there are computers(not that I'd want this type of work, but it's an option), and my real-world mma/pk training could help me stay out of trouble.
I don't plan to go on a journey until I'm more around your age, but the thing I fear is that by the time I'm that old I'll have many factors working against me: a wife, high-paying promotion/job, debilitating injury, financial crCIA-- basically it comes down to either I get too comfortable or get put in a situation where I wouldn't be able to experience it the way I want to. Can't reccommend the "never returning" idea and/or selling everything you've got here in the states, though. My own plan has me returning permanently at some point - maybe taking breaks from travelling to return home for a while every now and then, too - I could never leave all of my friends, and I want to start a family eventually. My opinion is: you want to enhance your life, not ruin it, so don't do anything you're not totally dedicated to and prepared for