The asylum seekers, whose wooden boat made it to within 500 metres of the Cocos Islands on Monday, were flown to Colombo in a highly-secretive operation under the cover of darkness on Thursday night.

The government has refused to comment on the forced returns, with a spokeswoman saying only: "we do not comment on operational matters".

Labor's immigration spokesman Richard Marles said this was not good enough.

"The Government needs to provide an assurance it has not returned anyone to harm," Mr Marles said.

"Australia has international obligations that this Government simply can't ignore. It is not good enough to try and dismiss this as an operational matter."

The small boat was intercepted close to the Indian Ocean archipelago, about halfway between Australia and Sri Lanka, in rough weather on Monday morning.

Then, local eyewitnesses said, it was carrying an estimated 12 asylum seekers.

But on Friday morning another local witness, who asked not to be named, said there were more than 12 asylum seekers on board, and said the group included women and children - including at least one infant.

He said the asylum seekers were transferred from the Ocean Protector customs vessel on to a smaller boat, before being taken to West Island, where they were loaded onto a bus.

Australian officials covered up all the windows, in an attempt to shield what was happening from a gathering group of locals.

But at least one local photographer was able to capture what transpired. It's understood he is negotiating to sell his photographs.

The asylum seekers were then loaded onto an Australian charter flight.

The wooden vessel is understood to be the first boat in about two years to make it so close to the Cocos Islands, although the government has turned back at least one other boat in the past year.

The government has refused to make any comment about the boat's approach, what process officials undertook to process their claims for asylum if any, or the operation to return them to Sri Lanka.

However flight records show a plane with registration identical to the plane photographed by locals on the Cocos Islands' airport on Thursday night departed Cocos Island for Colombo, the Sri Lankan capital, overnight.

The asylum route to the Cocos Islands opened up in earnest in 2012, when Sri Lanka's senior envoy in Canberra confirmed his government recently stopped a boat carrying more than 110 people departing for Australia via the Indian Ocean.

Before this, people-smuggling syndicates had not historically targeted Cocos Islands, preferring to send boats to Australian territory closer to Indonesia – either Christmas Island, south of Java, or Ashmore Reef off West Timor.

The distance between Ashmore Reef and the Cocos Islands is more than 4000 kilometres, a massive expanse to patrol.

The principle of "non-refoulement" is key to the 1951 Refugee Convention, to which Australia is a signatory. Under that principle, countries must not return asylum seekers to danger, and must properly assess asylum seekers' claims for protection.

The Gillard Labor government introduced so-called "enhanced screening" measures for Sri Lankan asylum seekers, which were later extended to other asylum seekers intercepted at sea.

This allowed authorities to conduct fast-tracked assessments on board the boat. Reports emerged in 2014 that this had occurred via teleconference, with asylum seekers being asked four basic questions: their name, their country of origin, where they had come from and why they had left.

Critics, including the UN refugee agency UNHCR have raised serious and repeated concerns that so-called "enhanced screening" measures do not offer potential refugees any meaningful chance of applying for protection.

In 2014, after Australia returned 41 Sri Lankan asylum seekers intercepted at sea, the UNHCR said the agency's "experience over the years with shipboard processing has generally not been positive".

"Such an environment would rarely afford an appropriate venue for a fair procedure," it said.

Mr Marles said the government should outline what process was undertaken to determine the refugee status of the people on board the boat.