Of course, I knew what he meant. He meant that Jews, in general, were mean. But I asked him what he meant and he thought I didn't understand his bigoted slur. So he explained. In detail.

I threw him out of my house. I would have allowed his partner to stay but she took his side. She thought I didn't have a sense of humour and I thought she didn't have any moral fibre. Perhaps we were both right. Having a sense of humour is overrated.

Whenever I recount this story – not so often these days – people are shocked.

But that's precisely the kind of conversation Tim Soutphommasane highlights in his new book I'm Not Racist But ... – it's casual racism. As he writes: "You do not need to be someone who advocates violence or noxious doctrine to do something with racist implications."

The effect doesn't depend on someone being, as Soutphommasane says, "a dedicated full-time bigot."

You can be a part-time bigot and still have an effect.

Anti-semitic abuse is rarer, these days, although not entirely absent from casual conversation. Instead, those who really think that way are likely to find each other on social media, where you'll find free-flowing bigotry and racism these days, in much the same way as you might have found it if you were attending the Reclaim Australia rallies over the weekend.

Which is why it was good to read that only a bare handful of nutjobs went to those rallies. They were joined by a police officer in Victoria who appeared to give a solidarity handshake to a Reclaimer, although police issued a statement which said: "The image of a police member engaging in a hand gesture with one of the demonstrators on social media at Saturday's rallies in the CBD represents a split-second action in a dynamic environment."

The Reclaimers were also joined by Coalition MP George Christensen in Mackay who said: "Our voice says we will not surrender, we will not sit idly by and watch the Australian culture and the Australian lifestyle that we love, and that is envied around the world ... we are not going to see that surrendered and handed over to those who hate us for who we are and what we stand for."

Hey, George, do you know that Muslims make up only 2 per cent of the Australian population? Do you seriously imagine that they'd have any chance against the Catholics? Good to know that you thought it was sad neo-Nazis who attended the rallies in Sydney and Melbourne – but you are enabling their bigotry. You and Pauline Hanson. Hanson, who appeared in Rockhampton and said that Australia was changing. "We have other different religions that have never been a problem in Australia," she said.

And as Soutphommasane said on Monday about Christensen: "Every member of parliament is free to accept invitations to speak but members of the Australian public will make their own judgments.

"These are groups which have very unsavoury elements within them."

It's always been the Right which sows racism, spreading the message which demonises the different and the new. Every time we get a new wave of migrants, the backlash comes; and with it, the desperate desire to link newcomers to something scary. With Muslims (who've been here since the first Afghan camel drivers arrived) it's Islamic State. The Vietnamese were linked to heroin.

Dai Le, businesswoman, aspiring NSW Liberal Party candidate and a local councillor, tries to be sanguine. She arrived with her parents as a refugee from a refugee camp in Hong Kong in 1979.

She says we always see an increase in media attention when there are new arrivals, particularly those newcomers who look different. But she says Australians could do with good leadership around race.

"The leaders in our community, especially the political leaders, have not managed race relations very well. It's got to be leadership overall, taking the community with you.

"There has to be a measured message as opposed to fear mongering. No one is stepping in to calm it down."

In a few months we will know for sure whether the government has succeeded in fomenting community unrest. Every year, the Scanlon Foundation publishes a map of social cohesion, supervised by Andrew Markus, a research professor at Monash University. He said that last year, the level of negative feeling about Muslims was running at about five times the level of negative feeling about Christians and Buddhists, which he believes is more of a reflection about international events than local events. And if that's the case, then it's unlikely to be sorted in this generation. Or even next.

I ask him whether he thinks any Australian leader has been able to promote social cohesion and he pauses. Then he says Bob Hawke.

"We just don't have leaders who have that level of support – he inspired confidence."

And none of our politicians do that. I want to reclaim Australia from them.

Twitter @jennaprice or email jennapricejournalist@gmail.com