That hasn't changed today, but our security culture is unrecognisably different. Almost all Commonwealth employees, and many private-sector contractors, hold a security clearance. It doesn't matter what their job is: public relations, payroll, plumber – without a clearance, they'll be shut out of the public service.

In itself, this isn't a problem of exclusion, as the bar seems rather low. The Defence Department's vetting agency knocks back fewer than one in every 830 applications it assesses. And half of the very few who are knocked back get a clearance anyway – just at a lower level than they sought.

But there are deep problems in this booming area of government activity; signs that the entire approach is wrong-headed.

To begin, let's go back two decades to one of the first public reports on how vetting was working (and not working). Back then, the issuance of security clearances was decentralised and highly inefficient, so much so that, according to then auditor-general Pat Barrett, the government had no idea how many people held an active clearance. Nonetheless, his audit found agencies issued about 11,500 clearances in 1998-99, and the number was rising quickly.

Among Barrett's concerns was that a "high proportion of staff had security clearances above the level that their work commitments would require". Similarly, every workplace he audited had classified files wrongly, with a clear tendency to overclassify them. This had the effect, he reported, of "increasing the costs of protection and restricting the flow of information" in government.

Yet he had a bigger worry: "a number of staff had access to information for which they were not appropriately cleared". The reason, he said, was that staff – especially temps and contractors – were being given access to classified documents before they were vetted because of the "long lead times to obtain clearances, commonly up to three months" (my emphasis).

All the signs today suggest Barrett's warnings were ignored and the problems have worsened.

When I wrote on this topic four years ago, the Australian Government Security Vetting Agency said about 350,000 people held active clearances. Its workload was about 33,000 assessments a year – triple that from 15 years earlier.

Last year, 46,500 applications were assessed. The total number of active clearances in circulation had climbed above 440,000.

What on earth is going on? The impression given is that, in order to protect its "treasure" (national security information), the government keeps building higher, wider castle walls and expanding them across its entire fiefdom, when it could simply hide its treasure better.

The reason the vetting agency exists is that, back in 2010, the Gillard government was so concerned by the security checks' rising costs that it centralised the process to eliminate waste. Money was poured in. The agency has unquestionably improved the efficiency of vetting practices.

But the main problem – lengthy delays – has worsened. Applicants wait, on average, 18 months for a positive-vetting clearance (which is needed to access information classified above top secret). Negative-vetting level 2 (needed for top-secret information) takes over five months, and negative-vetting level 1 (for secret documents) takes about 3½ months. Remember: two decades ago, government agencies were panicked about having to wait three months.

A crisis is now erupting. The Inspector-General of Intelligence and Security, Margaret Stone, complained recently to the Senate that she struggles to hire staff because recruits simply won't wait 1½ years to start a job.

Government agencies are so frustrated that many ask their recruitment agencies to offer interviews only to candidates who already hold a clearance; an unlawful practice that the Informant documented throughout last year. Who can blame them? With the ongoing staffing cap forcing agencies to rely on temporary employees, it's hardly reasonable to wait many months before they can begin their work.

Last year's independent intelligence review warned that the vetting delays are "exacerbating the intelligence community's existing workforce challenges". Its recommendation, unsurprisingly, was to give ASIO more money and staff so as to speed up the background checks. In other words, its proposed solution is to fix the supply of clearances.

Yet surely that's not the problem. It's easy to blame the vetting agency – perhaps it does need more resources – but it's today's ridiculously inflated demand for clearances that is causing and prolonging the backlog.

I've been told of least two offices in Canberra so poorly designed that everyone who enters must hold a negative-vetting clearance (level 1 for one office, level 2 for the other) – not because all staff need to access secret or top-secret information but because they must pass through a secure area to get to their desk. Somehow, getting clearances for everyone is seen as a better solution than thoughtful information security.

Overprotective classification has the same effect. When a workplace, over years, wrongly classifies its documents – a practice Barrett said was widespread – new recruits must apply for higher-level clearances than they really need, just so they can access basic information relevant to their job. The costs to everyone start to mount.

What would happen if the government decided to tighten security rather than expand it? To demand that public servants spare classification for information that genuinely warranted it, and to vet a minority of staff, rather than everyone?

It'd certainly save a lot of money. And the work of government would go on. Last year, the Independent National Security Legislation Monitor, James Renwick, told Parliament he didn't have the positive-vetting clearance needed for his job; he was still waiting with all the others in the queue (he did, however, have negative-vetting level 2). When asked whether lacking the right clearance was impeding his work, he laughed it off, explaining that "agencies have had no difficulty providing me with anything I have asked for".

If the watchdog of our national security laws can get by, I'm sure most public servants can, too.

There is a quick fix for at least part of this mess. Three years ago, Barbara Belcher's report on red tape recommended scrapping the default practice of new staff getting a baseline clearance, which involves checking their identity, qualifications, criminal history and background (going back five years). Replacing these clearances with a simpler form of screening would, Belcher said, "reduce the regulatory burden on staff and improve security outcomes".

Nothing has happened, though there are murmurings the government may finally move on this. On the other hand, the intelligence review ignored the suggestion entirely.

As always with issues related to national security, few politicians or senior officials are willing to be known as the one who asked: "Maybe we should do a little less?" It seems we can never spend too much on counterterrorism or counter-espionage, regardless of whether the costs justify the benefits – a point that academics Mark Stewart and John Mueller have demonstrated convincingly.

The great folly in all this, of course, is that officials with the highest level of vetting were responsible for the West's biggest known intelligence leaks. Edward Snowden and Bradley (now Chelsea) Manning are the best known, but let's not forget Canberra's own contributions.  There was Jean-Phillipe Wispelaere, a steroid-abusing officer at the Defence Intelligence Organisation who stole almost 1400 mostly US classified documents, flew to Bangkok and tried to sell them to a foreign power. And Simon Lappas, a depressed analyst at the same organisation, who fell in love with a prostitute and gave her documents to sell to the Russians. The Russians (at least, it was reportedly them) didn't buy.

The bureaucracy and the intelligence community's bizarre response? Just keep vetting more and more staff. How letting everyone inside the tent reduces the "insider threat" is beyond me.

Markus Mannheim edits the Informant. Twitter: @MarkusMannheim