Papua New Guinea authorities have moved to force the deportation of at least one asylum seeker housed on Manus Island.
It is understood PNG police officers arrived on the island in the early hours of Thursday morning.
They took one man to an aircraft while another ran away.
Federal Attorney-General George Brandis told parliament on Thursday he had been advised the men were not refugees, but Greens senator Nick McKim said their appeal processes had not yet been exhausted.
"I am advised that every single individual the subject of these removal proceedings is a person whose claim to refugee status has been processed and rejected," Senator Brandis said.
"It is quite wrong, therefore, for Senator McKim to claim as he does that rights are being violated."
PNG had every right in international law to return the men to their country of origin.
"That is what the New Guinean authorities are doing," the attorney said.
Senator McKim said the deportations should be stopped.
"Forcibly deporting them trashes their human rights, and places Australia and Papua New Guinea in flagrant breach of their international human rights obligations," he said.
"It is incumbent on the Australian government to ensure that the legal rights of all detainees are being protected, including all avenues of appeal."
The men's lawyer, Ben Lomai, said at least one Nepalese asylum seeker had been removed from the Australian-run Manus Island centre so he could be flown to Port Moresby and then to Nepal.
Mr Lomai told the ABC the government should not deport the men until problems with the assessment process were resolved.
"It's a concern that some of them ... may not have been assessed properly," he said.
Behrouz Boochani, an Iranian detainee on Manus Island, told the Huffington Post about 10 police officers arrived early on Thursday morning and woke two men.
"They took one of them while he was crying and the other one escaped and now is lost," he said.
The Nepalese detainees had been under pressure for some time, with five of them signing up to a $20,000 deal in exchange for them returning home.
"Immigration has said to the people with negative status sign this paper and get $20,000 or we will deport you by force," he said.
A Department of Immigration and Border Protection spokewoman declined to comment on Thursday.
"Questions relating to operational matters at the Manus regional processing centre should be referred to the government of Papua New Guinea," she told AAP.
A spokeswoman from the Department of Immigration and Border Protection told AAP the PNG government had previously confirmed any centre residents who had been found not to be refugees have no lawful basis to remain in the country and must leave.
"Substantial assistance packages are available to help non-refugees depart voluntarily, return home and re-establish their lives in their home country," the spokeswoman told AAP.
"In cases where non-refugees refuse to depart voluntarily, the government of PNG has indicated that it will enforce the removal of those individuals, in accordance with normal international practice."
