What started as a series of legitimate grievances by ordinary people, he says, has become "a combination of lunatics and rebuttalists and internet vigilantes".

"It just comes down to hating anyone with a different opinion," he says. "I had a hand in getting it to where it is, and I have some regrets about that...they've just been overrun by extremists."

While it's small, the ADL is one of myriad groups, including Reclaim Australia, the Patriots Defence League of Australia and the United Patriots Front – formerly led by "Great Aussie Patriot" Shermon Burgess – that make up the burgeoning anti-Islam movement in Australia.

The speed with which the movement has gathered pace, and an increasingly militant attitude, in recent months has alarmed some observers.

Almost 30,000 people had "liked" the Great Aussie Patriot's page on Facebook before Burgess removed it on Thursday night in an apparent fit of pique, and more than 20,000 have "liked" the United Patriot's Front Facebook page.

So emboldened have they become that the UPF, as Fairfax revealed last month, now has its sights set on the Senate, planning to campaign at the next election on the right to bear arms.

But if this seems far-fetched, it's worth remembering that Senator Ricky Muir was elected on the Australian Motoring Enthusiast Party platform with just .51 per cent of the primary vote at the 2013 election.

Far-right groups have been around for a long time in Australia. Notable examples include the Australian League of Rights (which was founded in 1946 and became a national organisation in 1960), Australians Against Further Immigration (established in 1989, and now defunct), One Nation (established in 1997), the Australia First Party (founded in 1996), and the Citizens Electoral Council (1988).

In the 1980s and 1990s, Australian far-right groups' grievances typically ranged from protecting Australian manufacturing and industries, to opposing Asian migration and multiculturalism. Others – like the League of Rights – were more openly anti-Semitic. Those on the extreme fringe like Combat 18 (named for Adolf Hitler's initials being the first and eighth letters of the alphabet) made no secret of their racist ideologies.

But the protectionist reflex of far-right groups now has a new enemy in its sights – Islam. Until earlier this year, Burgess was the spokesman for Reclaim Australia. But in May he left Reclaim in acrimonious circumstances and formed the UPF with Cottrell, Rise Up Australia's Scott Moerland, Melburnian Neil Erikson, who has pleaded guilty to the anti-Semitic stalking of a rabbi, and others.

The group has seized on the proposed Bendigo mosque as a symbol for far-right discontent over Australia's immigration policies, fears of terrorism and multiculturalism, and a vehicle for gathering members and notoriety.

But until now, few have spoken out from inside these movements.

Despite its name, the Australian Defence League is small, but its online presence can be combative. The former insider has asked we don't use his real name for fear of reprisals.

The man walked away from the ADL about two months ago, he says, at the same time as about six others left. He says the ADL used to represent legitimate economic and political concerns but became – under former leader Ralph Cerminara's influence – more concerned with "hating everyone who wasn't you".

"Ralph just hates brown people," he says. "I hate to say it, but it's all about being brown with him."

But Cerminara – who initially tells me I've been "blacklisted" by the ADL and UPF for writing "too many left-wing articles that are biased" (Cottrell did not respond to requests for an interview) – laughs at this, saying his mother is Aboriginal.

Cerminara was arrested in December for his role in an alleged affray in Lakemba. "We didn't touch anyone," he insists. "We spent five weeks in jail for saying the Prophet Mohammed was a paedophile."

But Cerminara says after he got out of jail in January, he decided that with a young family at home, it was time to hand the leadership of the ADL to another man, Chris Rothwell.

Rothwell – who worked as a detention centre officer between 1999 and 2005, and spent another eight months working in Nauru last year – says the direction of the ADL will change to be more in line with the English Defence League, a combative street protest movement in the UK.

"We're not bullies, we're not bigots, we're not racists," Rothwell says, adding that his wife is Sri Lankan.

Neo-Nazis and skinheads have been pictured at their rallies, but the ADL, UPF and others insist they are not racist. "Islam is not a race" is a favourite catch cry.

While Rothwell talks up the ADL's future, Cerminara is glowing about the UPF: "Anyone who considers themselves a patriot should really join the United Patriots Front."

He supports the UPF's push to run candidates for the Senate at the next election, but can't bring himself to back the Australian Liberty Alliance, a group modelled on far-right movements in Europe, which gained approval from the Australian Electoral Commission in August for registration as a political party.

"They're sort of like elitists, and they don't really represent the average person," Cerminara says. "They're just so rude and they think they're better than anyone else. You can't be like that."

These internal gripes seem to be par for the course in the far-right movement.

Late on Thursday, Burgess posted an online video in which he announced he was handing control of the UPF to Cottrell, upset at his treatment by fellow "patriots".

Meanwhile, the former ADL member says the league has grown to be more about opposing the religion as a whole and "leftists".

ANTIFA, or anti-fascists, are the sworn enemies of the far-right, a cross-section of socialists and anarchists, anti-racists and small-l liberals.

The more militant among the ANTIFA brigade have been fighting pitched battles with the UPF and their supporters at rallies in Bendigo and Melbourne, with both sides vowing to turn up anywhere the other does.

This stand-off worries the former ADL member. "I'm hoping that reason will prevail and [the media] will see that both sides are as bad as each other now and the general public will realise the sense of lunacy," he says.

"I see this as escalating [and becoming] more and more violent."