Mr Roberts is one of 19 owners of asbestos-contaminated houses who didn't sign up to the government's billion-dollar buyback, and instead demolished the home himself.

It has cost him $765,000 and he says while he's lucky to have been able to reorganise retirement and other savings, he shouldn't have to wear the entire cost.

"It's not my fault that I've got a dud house, it's the government's fault, it's as simple as that," he said, pointing to compensation for his sister, who lost her home in the 2003 bushfires, and for others hit by natural disasters.

Mr Roberts' case illustrates the dilemma for Fluffy owners deciding whether to sell to the government and move, buy back their cleaned blocks, or opt out altogether.

He had a valuation done for his Giralang cul de sac house, at $550,000, about $320,000 of which was the land value for the 990-square-metre block. If he had sold to the government, he would not have made enough to buy a similar house in the same area, and he believes it would have been prohibitively expensive to buy back the cleaned block.

So Mr Roberts decided to go it alone. The first demolition firm failed to get government approval, so Mr Roberts lost his $18,300 deposit. The firm eventually taken on charged $86,300. The new house, under construction, cost $499,000. Interest on a loan and rent adds another $160,000.

The cost will wipe out his wife's superannuation when she retires in 2019 and her share of a house she will inherit, along with money invested in a pension scheme.

Mr Roberts said his home felt like his "country" and he could not bear to move, as he was worried a move would exacerbate health problems.

"I don't see why I should have to move out of a place I've lived in for 40 years and invested in emotionally, financially. It just meant too much to me," he said. "You're in a bind, if you want to stay on your property that's what you've got to do."

He points to the different treatment of owners who signed up to private demolitions before October 28 2014, when the scheme began. Twelve owners slipped in before the deadline, and for them, the government is reimbursing the cost of demolitions and paying market value for their houses (minus the land value).

In NSW, Fluffy owners have the same deal, the government buying and demolishing their houses but leaving them the land.

Mr Roberts and his wife, who have two adult children, built the Giralang home in the 1970s and had the Fluffy insulation installed, but he said he didn't realise it was asbestos and was told by the installer that it was rock fibre.

Alerted in the late 1980s, he collected fibres from a gap in the linen cupboard and sent them in an envelope for testing, discovering it was asbestos. After the 1992 clean-up, he assumed, like others, that "everything was hunky dory".

He has done a number of renovations since, including a new kitchen, family room, deck, and government-approved work in 1996 that involved removing an entire front wall to extend the living room, and in 2014 work on the laundry.

But Mr Roberts doesn't dwell on the health implications and is reassured by the minimal asbestos found during the demolition, and by the years he spent out of the house overseas on postings.

"You've got to be optimistic. If you worry for the rest of your life about dying of Mr Fluffy you would live the rest of your life in terror."

While other Fluffy blocks get a new 99-year lease, Mr Roberts has also been denied that option.

An Asbestos Taskforce spokeswoman said the government had decided against allowing private demolitions under the buyback because of the need to vacate houses immediately and the importance of a co-ordinated demolition for economies of scale and an equitable approach.

To date, 967 owners have accepted offers, with 20 more still considering offers. Of the 19 not taking part, some were demolishing privately; others were leaving the property to their estate.