How the ABS now keeps closer tabs on us

The prices that give us a good steer on inflation were falling, by 0.4 per cent.

That's the change that month in what the Bureau of Statistics calls the consumer price index "excluding volatile items". The items it excludes (because they are often affected by supply disruptions) are fruit, vegetables, and fuel.

Apart from that, the measure of prices I just quoted is the best monthly measure of the prices of everything that households buy in the proportions they buy them.

Not all prices were falling. The price of alcohol was up, the price of bread was down, the price of rent was up, and the price of tourist accommodation was down. It's only on balance (excluding volatile items) that prices fell.

This is the sort of thing we wouldn't have known about until just a few years ago. Up until late 2022, the consumer price index was calculated only four times a year, and even that was a herculean feat.

The bureau had to collect what used to be 100,000 separate prices for each of those four surveys - a huge number collected in person, either over the phone ("hello, can you tell me your current price for ...") or in stores via handheld devices.

The cost to the bureau, and the number of staff involved, was enormous - big enough to make a monthly measure impossible, as important as that would have been to a Reserve Bank that set interest rates monthly and needed a monthly read on inflation.

But in the last few years the use of supermarket scanner data, "web scraping" to collect online prices, and data feeds direct from the computers of rental agents and all sorts of other businesses have cut costs enormously, and increased the number of prices collected each quarter almost ten-fold to 900,000.

The bureau says the monthly index isn't as comprehensive as the quarterly index yet, but it will be by the end of 2025, at which time the bureau will use it to replace the quarterly index, delivering something of the same quality 12 times a year.

That's just one of the ways in which an explosion of previously-inaccessible data is transforming the way the bureau goes about its job and is set to make statistics that used to be only fairly reliable suddenly very reliable.

For more than half a century, every month since April 1961, the bureau has published an update on retail spending - how much we are spending on shops.

The survey used to be quite useful. Back when it started, we did more than half our spending in shops. These days it's only one-third, the rest is on services.

And the retail survey was always a pretty rough-and-ready way to find out what we spent in shops. Each month the bureau surveys about 700 large businesses and 2700 smaller businesses selected at random. It uses phone calls and paper forms.

Meantime, in part due to the national emergency created by COVID, it's been given access to something better. Australia's big four banks agreed to give the bureau deidentified card and transaction data to enable it to quickly get a handle on how much were spending early in the pandemic, and they've kept providing it.

It turns out to be very good indeed. It covers far more retail outlets than the retail survey ever did, as well as spending on services and spending overseas, and it divides spending into types based on the type of merchant.

It doesn't directly cover what we spend in cash, but there's a lot less of that than there used to be. I'll replace the retail survey from the middle of next year.

The mammoth monthly employment survey of 24,000 households remains in place, as do the doorknocks and face-to-face interviews that begin each household's eight-month stint, but alongside it the bureau is developing a far more comprehensive measure using payroll data submitted to the tax office.

While payroll numbers can't tell us everything the employment survey does (they can't yet tell us the hours people work and whether are looking for work) they cover millions of Australians instead of thousands, and come out weekly.

The bureau is doing the same sort of thing almost everywhere. For more than a century it has surveyed farmers to find out what they are growing. It's begun supplementing that with data from satellites and the machines used on farms.

The ultimate goal of all of these sorts of changes, gathered together under the banner "Big Data, Timely Insights" is to ask as few questions as possible. Why run a survey, when you can find out directly?

It's far harder than it looks. A lot of the so-called administrative data provided to banks and other organisations isn't sorted in a way that makes it useful. That's where the bureau is concentrating its efforts. The more it succeeds, the less it will need to bother us and the better the information it will produce.

In the meantime, here's an update on those inflation figures, the ones that come out monthly. In February, the consumer price index excluding volatile items changed not at all, meaning in that particular month, inflation zero.

Even better, adding the past six months together (and multiplying by two) gives you an annual inflation rate of 2.5 per cent - slap bang in the middle of the Reserve Bank's 2-3 per cent target band, suggesting things are moving in the right direction.

It's too early to declare victory over inflation that was at one stage heading towards 8 per cent, but at the moment the monthly figures show things moving in the right direction.

If the direction changes, the bureau will tell us, quick smart.