Most of the Tax Office's battles over audits are fought against the super-rich, who have clawed back more than half the $909 million they disputed between 2011 and last year, according to the National Audit Office.

In his submission to a parliamentary inquiry into the ATO's handling of disputes, the Auditor-General notes the Tax Office has improved its performance in recent years.

But with more than $10 billion in outstanding taxes caught up in disputes as of June last year, much more needs to be done.

The inquiry is being held as the Tax Office faces scrutiny over its performance and conduct of disputes and imposition of tax penalties.

Fairfax Media reported on Thursday on the case of Sydney's Gary Kurzer, who is suing the Tax Office for nearly $6 million, alleging its negligent and mistaken pursuit of taxes cost him his business, home, marriage and health.

Mr Kurzer and taxation lobbyists say thousands of ordinary people have been crushed in disputes with the ATO because they lack the money and the will to fight the government body.

But the Auditor-General has confirmed that for the "high wealth individuals" and those targeted by "Project Wickenby", who are nearly always backed by expensive legal and accountancy expertise, it pays to take on the tax man.

Project Wickenby has been the country's biggest tax investigation, costing more than $500 million and leading to scores of arrests including entertainer Paul Hogan.

About two-thirds of the objections to "amended assessments", which are usually tax returns that have been audited, have been mounted by high-wealth individual taxpayers and those covered by Project Wickenby.

The Audit Office found that lodging an objection was well worth a try – "positive outcomes" were achieved by 49 per cent of for high-wealth individuals and 37 per cent of Project Wickenby targets.

A sum of $477 million had to be handed back to high-wealth individuals between 2011 and last year, the office says.

The Auditor-General's inquiries found that the ATO's dispute resolution mechanisms sometimes worked well, but there was a high rate of disputes caused by its errors, taxpayers withholding information, lengthy waits for resolution "and consequent frustration by taxpayers about the responsiveness, cost and sometimes fairness of processes".

"These findings have highlighted scope for the ATO to improve the efficiency and effectiveness of dispute resolution processes, to the benefit of the organisation and taxpayers," Mr McPhee wrote in his submission.

There was good news for the Tax Office in the submission from the Commonwealth Ombudsman, who noted that 1369 complaints about the agency had been received in 2013-14, a decline of nearly a quarter on the previous year.

In its submission to the parliamentary inquiry, the Tax Office pointed to improved performance in several areas, including disputed debt cases with "significant reductions" of 31 per cent for small business cases and 41 per cent for individual disputes.

Commissioner of Taxation Chris Jordan wrote that a new approach to issues relating to high-wealth individuals was also showing results.

"We are already seeing very positive results in the large market and for private groups controlled by high-wealth individuals, with both parties willing to come to the table early and discuss issues openly and in good faith," he wrote.