Given the extent to which governments rely on private contractors for a large range of goods and services, it is unsurprising that contractors are often in the frame when things go wrong. But many of the recurring issues arise out of factors specific to the contracting process.

A common complaint is that the use of contractors has caused agencies to run down their in-house expertise and technical resources. As a result, it can be argued, agencies lack the capacity to assess whether the contracts they are agreeing to give the government and taxpayer adequate value for money. Without their own professional judgment, grounded in technical knowledge and practical experience of the area in question, public service managers are ill-equipped to decide matters of all-round quality. Instead, they tend to fall back on generic checklists of assessment criteria that emphasise easily specifiable factors, such as cost and timeliness.

Alternatively, if funds allow and time permits, they may contract in an external consultant or commissioner to give an expert opinion on a proposed contract. But such advice carries the risks of perverse incentives attached to all forms of external contracting and consulting. The consultants' objective is to gain future contracts, which encourages them to say what they think governments want to hear in preference to what they ought to hear. Once again, without the professional judgment to tell the difference and without their own in-house, trusted staff to advise them, public service managers are at the mercy of self-interested outsiders. This vulnerability is compounded by the lack of transparency that surrounds contracting. Overuse of commercial-in-confidence provisions has shielded public servants from the bracing effects of public scrutiny.

The lack of in-house capacity can affect not only the initial process of drawing up contracts but also the oversight of how contracts are implemented. Contracts require payments to be made in return for agreed outputs and outcomes, which the contracting agency must verify. Some contracts are easily administered; for instance, procurement contracts for identifiable pieces of equipment or service contracts for straightforward services, such as cleaning and transport. But others, particularly contracts for providing social services, require the achievement of more subtle outcomes that are qualitative and contestable in nature. For these services, governments have turned to "relational" contracts, which aim to create partnerships based on shared values and trust. In this way, contractors are relied on to exercise their professional judgment and pursue the public interest out of their own sense of commitment to the joint enterprise. Classic examples have been the contracting of church and community-based organisations to provide valuable social services at reasonable cost.

However, as the constant stream of scandals illustrates, many commercial providers are more interested in profit than in good service and cannot be trusted to do the right thing. As a result, governments are being driven to impose tighter controls and regulations. Even then, in the face of determined rorting and corner-cutting, most government agencies lack the resources to prevent opportunistic contractors from wrongfully expropriating public funds. Meanwhile, trustworthy contractors in the non-profit sector are being overwhelmed by intrusive and inappropriate controls.

After two decades of wholesale outsourcing, some general conclusions are clear. Contracting out is an efficient and effective alternative to in-house provision where the objectives are clear and easily monitored, and where there is a competitive market of alternative providers. It also works well for more complex services where providers can be trusted to pursue public-interest objectives for their own reasons.

However, where these conditions of either simplicity or trust do not apply, the risks that governments will not receive value for money start to build. Moreover, extensive experience with outsourcing has itself compounded these risks by reducing governments' capacity to effectively draw up and monitor outsourcing contracts. Some complex service contracts that could have been safely contemplated a generation ago are now beyond the professional expertise of public servants to administer.

None of this surprises public administration academics. It is more than 20 years since Rod Rhodes wrote a seminal article in Political Quarterly titled "The hollowing out of the state: the changing nature of the public service in Britain". Australian scholars took up the concept of the hollowed-out state as a warning about the effects of contracting-out. Other academics around that time drew attention to the difficulty of specifying objectives for complex services and to the accountability deficit resulting from transferring public services from the public to the private sector. But, like Cassandra in Greek mythology, who was cursed with the power to predict the future but never to be believed, public administration experts were ignored in the worldwide enthusiasm for privatising the state.

Where to from here? An interesting contribution has come from Gary Sturgess, Australia's foremost intellectual advocate of privatisation and outsourcing. Sturgess has had a varied career, working for governments (notably the Greiner government in NSW) and as an author and consultant. He also spent a decade in Britain with Serco, the multinational company specialising in service contracts for public organisations, such as prisons and hospitals. He holds positions with the Australia New Zealand School of Government, the University of NSW and Griffith University.

Sturgess recently published a report for Britain's Business Services Association, which represents major service providers in Britain (Just Another Paperclip? Rethinking the Market for Complex Services). He identifies a crisis in Britain's outsourcing services industry, reflected in the cancellation of several high-profile contracts and the placing of other providers on notice. Some major contracts are losing money and, among five major contractors, only one has made a commercial return in recent years. Many companies are refusing to tender and are leaving the British market. Companies also complain about excessive controls imposed by governments and about the frequent turnover and lack of professional experience among the government counterparts with whom they deal. Contractors have taken over from government agencies as the custodians of corporate memory.

Britain has long been at the international forefront in privatisation and outsourcing, and its experience is therefore relevant to Australia. Sturgess's main diagnosis is that the British government has lost sight of the key differences between the varying types of service they are buying. In particular, they are forgetting that complex ongoing services cannot be adequately dealt with by transactional contracts but require trust-based relational contracts. By treating all contracts as transactional in form, they are reducing the purchase of social and other complex services to the level of buying office supplies ("just another paperclip"). According to Sturgess, this is a relatively recent development, replacing an earlier reliance on trust-based partnerships in the outsourcing of complex services.

Sturgess's solution is to concentrate on restoring trust between governments and their long-term private providers. Governments should give up trying to impose ever tighter controls on contractors. They should also avoid a race to the bottom in always preferring the lowest-cost tender regardless of proven record. Instead, they should be more willing to enter into ongoing dialogue with service providers with whom they can establish good working relationships.

Admittedly, Sturgess is writing for the established service providers and naturally reflects their interests in eliminating cut-price rivals and excessive government oversight. He glosses over the extent to which governments have been provoked into tighter controls by unscrupulous contracting partners abusing their trust. He is also spruiking the role played by designated contract commissioners in brokering relation between contracting parties, a role in which he himself has specialised. Even so, his analysis deserves close attention.

What is remarkable is how far Sturgess agrees with other longstanding critics of outsourcing who come from a standpoint that is more sceptical about the private provision of public services. Like them, he argues that government agencies have become dangerously hollowed out and are unable to act as effective public partners in complex outsourcing partnerships. Like them, he accepts that policy objectives often cannot be reduced to simple measurable indicators and targets. Like them, he also holds that mutual trust between governments and providers is crucial for effective service provision.

The key difference is that, whereas the sceptical critics are only prepared to trust providers from the public and non-government sectors, Sturgess also has confidence in commercial contractors – provided certain conditions are met. This approach is more likely to appeal to a Coalition government, with its close links to the business sector and its openness to lobbying from major international contracting firms.

Conservative politicians need to recognise that outsourcing complex government services requires the development of trust between the parties, which means looking beyond the short-term bottom line and not always preferring the cheapest option. In addition, successful contracting depends on well-resourced government agencies with the skills and experience necessary to manage ongoing relationships with contractors. Running down government staffing levels while relying more on private contractors is a recipe for continuing policy failure.

Contractors, for their part, must earn the right to be treated as trusted partners. They must be prepared for the long haul and willing to learn from experience. They must also be ready to submit to the level of public scrutiny and accountability that public servants take for granted. Indeed, given that the commercial private sector is not imbued with the same commitment to serving the public interest, there is a case for subjecting private contractors to more scrutiny than the public service, not less.

Richard Mulgan is an emeritus professor at the ANU's Crawford School of Public Policy. richard.mulgan@anu.edu.au