The commission – full title, the Royal Commission into Misconduct in the Banking, Superannuation and Financial Services Industry – has, during its investigations, considered regulators including the Australian Securities and Investments Commission, which regulates corporations and the financial system (including banks), and the Australian Prudential Regulatory Authority, which supervises financial institutions to ensure they can meet commitments (for example, banks are able to give account holders their money on request, insurance companies have enough funds to pay valid insurance claims).

They, together with the Reserve Bank and the Treasury, come together in a Council of Financial Regulators. The council says it has two roles: "to contribute to the efficiency and effectiveness of financial regulation" and "promote stability of the Australian financial system". However, the entries on its website suggest threats to financial stability predominate, with regulation effectiveness coming a distant second.

The council's limited role in promoting good regulation was highlighted by a draft Productivity Commission report earlier this year, which found "competition in Australia's financial system is without a champion among the existing regulators". "In a system where all are somewhat responsible, it is inevitable that (at important times) none are".

The Productivity Commission's final report on financial system competition was released last Friday. Although the language moderated, the fundamental messages remain. The report says "while there is much hand‑wringing about competition, there has been little shift in the regulatory culture. The emphasis on stability – best represented by the repeated use of the phrase 'unquestionably strong' – persists ... there is no counterweight that can advance the benefits of competition and possibly influence regulators before they intervene in the financial system. Discovery of alternatives relies entirely on in‑club thinking ..."

Treasurer Scott Morrison released the report at an address to the Australian British Chamber of Commerce. He said "we need better disclosure. Banking and financial regulation has had a dulling effect on customers ... inertia, similar to what we have seen with households and their electricity and gas providers and super funds, is placing the clamps on genuine competition in the financial sector. And the big players know it."

In short, regulators who administer highly complex business rules and processes benefit big players (who can afford big legal fees) while ordinary Australians suffer. We saw that with regulation of energy markets (now being reformed) and currently with finance. There should be an onus on regulators to aid transparency and competition as part of regulatory practice. Limited competition, while it might make large companies more stable, harms customers. While competition does not guarantee good behaviour, it makes misconduct much less likely; institutions have more incentive to treat customers well, and ethically, so as not to lose them to competitors.

Insufficient competition is, however, only part of banking and finance industry misbehaviour. The evidence presented to the royal commission suggests something more fundamental is at play. If all we were seeing were financial regulators, perhaps fearful of another global financial crisis, propping up large banks and insurance companies at the expense of customers, then – sad though this would be – it would have few lessons for other regulators.

The more important lessons arise from the evidence that has emerged about how the regulators interact with the financial institutions they should be regulating. Royal Commissioner Kenneth Hayne has queried why ASIC, especially, appears to take a soft line with banks rather than cracking down on misbehaviour.

Similarly, the senior counsel assisting the commission, Rowena Orr, QC, has asked searching questions about what are known as "enforceable undertakings" reached between ASIC and the limited number of financial institutions ASIC has caught doing the wrong thing. Under questioning, it has emerged these agreements are almost always negotiated down from the desired penalty, and have little practical effect. Moreover, by contrast with the open processes involved in a court action, enforceable undertakings are negotiated behind closed doors, in secret. An enduring lesson in governance is that secrecy encourages misconduct, while transparency encourages ethical behaviour.

Policymakers who design regulatory institutions have for decades been aware of the problem of regulators being "captured" by the industries they regulate. Indeed, economist George Stigler won a Nobel prize for his work on the concept in the 1970s. Norman Abjorensen provided an excellent outline of the theory of regulatory capture and its application to Australian banking in June's Informant.

Most Australian regulators are well aware of the risk of capture and try to avoid it. Direct and overt capture – for example, companies providing payments to regulators in exchange for favourable decisions – is thus rare. Unfortunately, a much more common form of "soft" capture arises as a result of well-intentioned but fatally flawed appointments by successive governments of regulators who "understand" their regulated industries. Often, this is at the urging of industry lobbyists, who tell ministers it is essential for regulators to have a thorough understanding of the industry concerned – and the only way someone can have such an understanding is to have previously worked in that industry. This is pure self-interest. Even if, as Abjorensen suggests, "detailed knowledge of an industry is essential for a regulator", that knowledge can come from careful study, hiring good advisers, consultations or investigations; working inside an industry is not the sole or even necessarily the best way to understand it. Insiders often overlook the worst features of their own industry, seeing them simply as "the way things are done round here".

Where an industry has systemic problems rooted deep in industry culture, it is even more important that regulators come from outside. Insiders are likely to be blind to the problems because they have been part of the culture and picked up its underlying values and assumptions. The evidence presented to date to the royal commission clearly points to cultural failings in banks and other financial institutions.

Anyone who has dealt with high-end lawyers and financiers will have observed a peculiar view among many – not all – that instead of "might makes right" it's "money makes right". Great wealth is seen as a sign of great morality. For many people in financial institutions, an unspoken but shared cultural assumption is that amassing wealth is inherently virtuous. Those of this view enlist the word "good" to unfamiliar uses: someone is described as a "good" banker because they make tonnes of money. That's profitable and successful, not good. Sometimes the banker concerned is a good person, sometimes not. There is no reason to suppose successful bankers are more moral than others. Indeed, the history of corporate failings – the rise and fall of Enron in the United States or the Volkswagen emissions-tampering scandal come to mind – suggests misconduct can be highly profitable, at least while it remains undiscovered.

If banking and finance regulators are drawn from a pool of people inside that industry, they will be more likely – not inevitably, but more likely – to ignore cultural problems. Despite this, successive governments have for the most part appointed lawyers and bankers to be our financial regulators – a sure prescription for application of the law but far more dubious for regulatory oversight of culture and attitudes. We need expertise in financial regulation, but we also need the balance that would come from appointing at least some regulators who are neither lawyers nor bankers.

The lesson for regulatory design more broadly is to appoint a broad spread of people to regulatory bodies rather than prioritise industry knowledge. That can come in time. More important is to have independent regulators without cultural preconceptions. One reason why the Australian Competition and Consumer Commission is a more effective regulator is its broad scope: members cannot be single-industry experts. It thus avoids the trap of soft capture through shared culture. Independent thinking is more desirable than specialist knowledge: an important lesson for regulatory design.

Stephen Bartos is an economist and a former Finance Department deputy secretary.