The world No.1 tennis player was detained at Melbourne's Park Hotel last week when his visa was cancelled, drawing attention to asylum seekers also being held at that venue on an indefinite basis.

But the Visa Cancellations Working Group, Asylum Seeker Resource Centre and Refugee Advice and Casework Centre say Djokovic's ability to even challenge - let alone successfully - his cancellation was "the absolute exception", with most detainees being denied procedural fairness.

"Most visa holders whose visas are cancelled on arrival are denied time to provide a response and the chance to get advice from a lawyer," VCWG deputy chair Sanmati Verma said.

"Many are turned around at the border quickly while in immigration clearance with little or no recourse to redress, even if they face serious harm in their home country.

"The laws currently in place require people to be mandatorily detained until they are either granted a visa or deported, resulting in many people being detained for years, even indefinitely."

Along with the "dysfunctional and dangerous" visa cancellation regime, advocates say the 47 personal powers available to the immigration minister is an inappropriate amount of power to override decisions made by courts and tribunals.

Powers through the Migration Act were expanded in 2014, which the bodies say led to a "huge increase" in visa cancellations that were "alarmingly prone to error and injustice".

The groups are calling for a federal inquiry focusing on the "restoration of substantive and procedural rights".

"Cancellations have life-changing consequences for visa holders, their families and communities - including permanent separation and return to countries where a person might face death or serious harm," a statement read.

"The damage to individuals and to Australia's international reputation is severe."

Australian Associated Press