First time I went to Asia, I was freaked out.

It all came to a head on a beach in Vietnam, just near Hoi An – nice enough spot, a huge arc of sand with plenty of room to stretch out. I'd got myself all set up, ready for an afternoon of peace and quiet away from the madness of the city's tailoring touts, when the first one approached.

"Mangoooooo! Peeeeeeeeanut!" There was an old woman walking over, arms clasping a basket of fruit. She stopped in front of me. "Mango?"

I shook my head, so she squatted down on the sand and went through each item individually. "Mango? Peanut? Pineapple?" After five minutes or so she wandered off, but that lady was to be the first of a conga line of touts who would push everything from beads to foot massages over the next few hours.

You gotta be kidding me, I thought. Aren't beaches sacrosanct? Isn't this the one place you can finally get away from the touts?

But it wasn't. And so I got annoyed with the touts, I got annoyed with the beach, I got annoyed with Hoi An and I got annoyed with Vietnam. And it stuck. Any time someone asked me about the country from then on I'd say, "Yeah, it's all right, but the constant hassle ruins it".

Like I said, first time in Asia. And the first time you're subject to that sort of attention, it can freak you out, become frustrating, even frightening. Everyone wants a piece of you; everyone approaches you, it's impossible to tell the good from the bad.

I'm writing this, of course, in response to fellow Fairfax writer Carolyn Webb's comment-monster from yesterday, detailing her new-found aversion to Bali. Personally I thought most of it was tosh – hating Bali because of the touts is like hating a music festival because it's too loud (what did you expect?) – but I can understand how she came to feel that way.

My guess is it's the first time she'd been to Asia (or at least strayed from five-star-resort Asia), and was a bit shocked. Touts' continual hassle becomes a nightmare when you're not used to it; jumping on the back of a motorbike taxi sounds insane if you've never done it.

(But really, you want aggressive touts? Try Cairo, or Rajasthan, or Siem Reap, or Marrakech. They'd eat a Balinese taxi driver for breakfast.)

If you followed the comments yesterday you'd have seen a fair few people sunk the boot in, which is all fun and games on a travel blog apparently, but there was one particular strand of comment that stood out for me: the "people like you should stay at home" types.

There were loads: "don't leave your house again"; "people like you shouldn't be allowed to travel"; blah blah blah.

Leaving aside the travel elitism, the implication that if you don't fit a certain mould you shouldn't be allowed to have a passport, that sort of thinking is completely flawed. In those people's minds, inexperienced travellers shouldn't be allowed to travel.

Seriously? If anything, inexperienced travellers should be encouraged to travel more. Give 'em all passports and one-way tickets, send 'em on their way.

Because I guarantee, the more you travel through foreign countries, the better it gets, and the more you come to appreciate the differences rather than be freaked out by them.

(Yeah yeah, I've been accused of being a whinger who should just stay at home too. I try to travel with the mindset that "it's not wrong, it's different", but that may not always come across.)

The more you travel, the easier it is to deal with touts. They can still be annoying, but if you expect to see them and know how to treat them, it's a much more enjoyable experience. You laugh with the friendly ones, brush off the aggressive ones.

The more time you spend on the road, the less likely you are to be shocked by motorbike taxis, or, say, squat toilets, or weird food, or the language barrier, or... wooden penises.

My guess is that with a bit more experience, Carolyn would have a much better time in Bali, or anywhere in South-East Asia. Maybe she'd even take a motorbike taxi – it's pretty good fun.

I eventually returned to Vietnam about seven years after my first visit, even braved those beaches, and I couldn't figure out what I'd been so annoyed by the first time around. I loved every second of it. And the difference was experience.

Have you found you've enjoyed travelling more with experience? Are there things that used to shock or annoy you but now seem run-of-the-mill? How do you deal with touts?

Contact Ben Groundwater through his website, follow him on Twitter, or email bengroundwater@gmail.com