But the trend is international and crosses party lines. The OECD's latest Government at a Glance report breaks down government production costs – i.e. the money governments spend buying goods and services. On average among OECD nations, 44 per cent of this spending in 2011 was on outsourcing, compared with 47 per cent on government employees. On average, outsourcing represented 10 per cent of gross domestic product, a reminder that outsourcing may reduce the size of the public service but not necessarily the level of public spending.

Does this large transfer of taxpayer funds into private hands provide the public with value for money? Can we be assured that all such funds are directed to proper public purposes?

These longstanding issues have come to prominence again in recent discussions over "the Senate order", which, since 2001, has obliged government agencies to table in Parliament a list of all recent contracts worth over $100,000. As well as listing the name of the contractor and the contract's purpose, agencies must also list any provisions for confidentiality in the contract and the reasons for such requirements.

To monitor the order's reliability, the auditor-general has been required to report each year on the use of confidentiality provisions in a selection of government agencies. The latest such report was released in September. Whether, and in what form, the Senate order should continue was the subject of a Senate finance and public administration committee inquiry earlier this year.

Contracting out has always presented challenges for public accountability and transparency. Certainly, where objectives can be clearly specified, governments can hold contractors to account for achieving the desired results. However, many contracts are open-ended and relational in form, requiring ongoing consultation and the evolution of objectives over time. In addition, private contractors are not subject to the levels of accountability for matters of process faced by public organisations. They do not normally face direct scrutiny – for example, from parliamentary committees or government auditors. More generally, both private companies and the public servants who deal with them can use the principle of commercial confidentiality to prevent the disclosure of information related to contracts.

In the early years of the modern outsourcing movement, outsourcing advocates argued that governments remained accountable for contractors' performances. Accountability could even be enhanced by specifying objectives. Alternatively, accountability deficits could be welcomed, sotto voce, as a means of circumventing the cumbersome regulations and red tape that hobbled public service performance.

The defenders of public accountability, however, fought back, seeking to extend their scrutiny into the hidden workings of the contracting process. A former federal auditor-general, Pat Barrett (1995-2005), was an outspoken critic of the limits placed on his office's access to contracts and of the excessive use of commercial confidentiality to hide details of contracts. He encouraged agencies to include clauses in contracts that would allow the Australian National Audit Office to audit certain aspects of contractual performance, such as accuracy of reporting and compliance with due process. This developed into the standard non-mandatory clauses for providing ANAO access to contractors' information, which are on the Finance Department website.

Barrett's advocacy was also reflected in the Senate order's focus on commercial confidentiality. The Finance Department, in its advice to agencies, lists four criteria that commercial information must meet to be legitimately counted as confidential:

Applying these criteria, together with the ANAO's annual reporting and monitoring, has clearly reduced the reliance on commercial confidentiality in government contracts and correspondingly improved transparency. The current Auditor-General, Ian McPhee, notes that the proportion of reports containing confidentiality provisions across all government agencies declined from 24 per cent in 2001-02 to 4.5 per cent in 2013. In addition, the level of misuse of the confidentiality criteria, as revealed by the ANAO's examination of sample contracts, though still significant, is mostly the result of misunderstanding the reporting requirements rather than genuine over-concealment.

McPhee himself, in evidence to the Senate inquiry earlier this year, said former practice had been to stamp almost everything "commercial-in-confidence" in a somewhat mindless fashion. He had noted the same attitude in his own organisation, citing its contracts with accounting firms. "We have gone past that," he said.

Indeed, McPhee considers that the Senate order and the annual reports have now achieved their main purpose and could be phased out, leaving the main burden of transparency to be carried by the Finance Department's AusTender website, supplemented by occasional ANAO performance audits. Finance Minister Mathias Cormann and his department agree, in the interests of simplifying regulation and compliance.

The Senate's finance and public administration committee, while sympathetic to reducing the compliance burden, has expressed concern about whether AusTender, in its present form, is capable of presenting the information given in the Senate order. In the meantime, it has asked the Auditor-General to continue to report on compliance with the order, but biennially rather than annually. In the committee's view, the gains in transparency elicited by the Senate order are not to be risked in favour of reducing red tape.

The committee's scepticism is fully warranted. The instinctive public service response – that any information concerning a private company is to be treated as commercial-in-confidence – is still well-entrenched and on full display in other contexts, such as dealing with freedom of information requests or answering questions at estimates hearings. The ground so painfully gained could easily be surrendered.

At the same time, more inroads could be made into the secrecy surrounding government contracts. One lesson from the last decade, particularly with large-scale public-private partnerships, is that big problems, such as cost overruns and time delays, are often due to faults in the initial contract that only emerge later. Governments have repeatedly signed up to contracts that supposedly share the risk with private companies but in which the government ends up carrying most of the risk itself. A major problem is bias in favour of the contracting companies on the part of those with the government's ear – not only well-heeled lobbyists but also consultants who often work for contractors as well as governments. Opening up the contracting process itself to independent scrutiny could often save the government and taxpayers from significant potential losses.

The Productivity Commission, in its recent report on public infrastructure, agrees. For example, it recommends that all cost-benefit analyses should be made openly available as part of the tendering process, allowing their assumptions to be contested and criticised publicly. It rejects the common view that such information should be kept secret on commercial-in-confidence grounds.

Britain's National Audit Office recently explored the transparency of government contracting with four multinational companies active in Britain (and also in Australia): Serco, Atos, G4S and Capita. Its report then formed the basis of hearings by the House of Commons' public accounts committee, in which company executives were questioned. One interesting conclusion was that the companies themselves were much more relaxed about disclosing information than were the government departments with whom they dealt. It was the public service, not the companies, that took the lead in imposing "commercial-in-confidence" embargoes. Even allowing for Whitehall's particularly strong traditions of secrecy, such a finding would probably be replicated in Canberra.

The British public accounts committee and National Audit Office also emphasised the value of "open-book accounting", whereby the books of each partner in a contract are open to the other. This practice is standard in private sector partnerships and alliances, and can be extended to public-private partnerships and outsourcing contracts generally. In a public-private context, access to such information would allow government auditors to assess, for example, the profits made by contracting companies and whether the costs charged to governments were reasonable. The House of Commons committee recommended that open-book accounting be applied to all government contracts above a certain value and that public servants be trained to use it to protect taxpayers' interests.

As McPhee has indicated, the search for transparency in government contracting goes well beyond enforcing the Senate order, though that has been of value. Many of the ANAO's performance audits concentrate on the operation of individual contracts, particularly in defence procurement. In 2011, after recommendations from the joint committee on public accounts and audit, the Auditor-General acquired the power to "follow the money" by auditing the performance of contractors who receive money for a Commonwealth purpose, a power he has yet to exercise.

By the very nature of their activity, auditors can only scrutinise after the event and cannot engage directly in drawing up any particularly contracts. Within these limits, however, they have emerged as the main champions of transparency in outsourcing. But changing the public service's culture of secrecy in contracting remains an uphill battle.

Richard Mulgan is an emeritus professor with the Crawford School of Public Policy at the Australian National University. richard.mulgan@anu.edu.au