Fords and Holdens – the cars beloved by generations of Australians – are driving up and down the back streets intersection, bounded by a Presbyterian church, a Uniting church, an Anglican church and a food charity.

Besides the unmarked Commodore police cars are older models, driven by people getting out and taking selfies in front of the signpost marking Mackenzie and Forest streets.

It is these people who Premier Daniel Andrews scornfully described as being unable to spell the name Bendigo, after meeting with local religious leaders in an attempt to ease tensions.

"You've got people who wouldn't know how to spell Bendigo coming to Bendigo, trashing Bendigo's good name," he told reporters on Friday. "They're outsiders in every sense of that word. They're outside the norm, they're outside our values, and they're outsiders to the great city of Bendigo."

The United Patriots Front and other far-right groups have called a day of protest, "Rise Without Fear", for Saturday, to coincide with the so-called "global rally for humanity", which calls on people to protest against Islam and Muslim migration at their local mosques, in co-ordinated protests around the world.

In Victoria, Bendigo has become a flashpoint for far-right discontent over a raft of issues, including Australia's immigration policies, fears of terrorism and halal food, and multiculturalism.

But most here believe the city is being used by outsiders for their own ends.

Heri Febriyanto, the spokesman for the Bendigo Islamic Association, will be staying well clear of the rally.

"It's nothing to do with us, because they're just using, and taking for granted our issue for their own agenda," he said.

"As we know, 99 per cent, I would say, of the protesters are from outside of Bendigo. I don't think the local people will be supporting this rally."

Simon, a Bendigo business owner who asked for his surname not to be used, said the anti-mosque protesters' names "are mud in this town".

Asked what the debate had been like, he said: "What debate? There is no debate. Just build the thing and get on with it. We're the pivot in the middle in all of this. There's one side, and the other side, and we're just in the middle and they're just using Bendigo for their own reasons."

Businesses several blocks in either direction of the protests have hired security guards, and some will shut their doors to avoid trouble on Saturday.

The potentially violent protests had forced the cancellation of the Bendigo Heritage Uncorked Festival for Sunday – which had been planned for nine months – infuriating organisers.

Bendigo mayor Peter Cox said he had no doubt the mosque would be built, and also blamed protesters from outside Bendigo for whipping up trouble in his town.

"We don't want a small minority dictating to the rest of Australia what the issues are."

Victoria Police Assistant Commissioner Stephen Leane said police were prepared to take "whatever action we have to take" to ensure the safety of the community.

With Rania Spooner