More than 800 men remain on Manus and are being pushed to relocate to its main town, despite their security fears.
Authorities say it is safe but concerns are high of further violence, both at the detention centre and in the town.
They're guarded moments, when refugees make the trip into Lorengau, the main town of Manus Island, from the detention centre at Lombrum across the island.
Sri Lankan Tamil Shamindan spent more than three years in detention before the doors were opened after the centre was declared illegal last April.
"Really it's not safe for me. I fled my country to protect myself because I was persecuted and tortured in my country and completely the locals don't want us to stay here."
Many other refugees venture into Lorengau regularly but numerous alleged assaults and robberies have left some wounded and many fearful.
Across the island at Lombrum, demolition of Foxtrot compound at the open detention centre has begun, and educational, health and recreational services cut for refugees.
As Australian and PNG authorities shut down the detention centre they're trying to force the refugees to move into Lorengau township to access services, but the men say they just do not feel safe here.
"I want to settle in a safe country rather than PNG."
A temporary transit centre near the main town houses dozens of refugees and could take hundreds more.
Chief Inspector David Yapu is the Manus Island Provincial Police Commander.
"I think the refugees should not be worried because of the kind of hospitality and friendly approach the local people have."
But the refugees want out.
Told they will never go to Australia, hopes for US resettlement are high.
Hundreds have been interviewed including Pakistani refugee Naeem.
"First of all we need safety and we are happy to go to America. We don't know when we will move to America. It is unclear."
Disappointment lingers for Naeem and other refugees over the $70 million out-of-court settlement for hundreds of detainees last month from the Australian government.
"Money cannot bring back my four years of life that I have lost without committing any crime."
After more than four years at the detention centre, for now they still see it as the safest place.
Behrouz Boochani is a Kurdish-Iranian refugee who recently premiered a secretly recorded film about Manus at the Sydney Film Festival.
"If the government wants to take people by force then definitely we will have some violence, maybe riots, because the people feel they don't have anything to lose."
PNG prime minister Peter O'Neill says the country wants to end this episode peacefully.
"In managing the last remaining people on Manus Island, we've offered them resettlement in the country as well, a few of them have taken that up, I think less than twenty in total. We are looking at all options."
The Manus detention centre is due to finally shut in October.
Australia is also offering refugees tens-of-thousands of dollars to return to the homes they fled, an offer that expires in August.
Sri Lankan Tamil Shamindan is unsure about what will happen to him and others in a few months' time.
"My life is in limbo, you know, I really have no choice. No choice."
