Participants work with data to uncover facts at the Melbourne Hackathon held at The Age building in Melbourne on February 4, 2013.

Participants work with data to uncover facts at the Melbourne Hackathon held at The Age building in Melbourne on February 4, 2013. Photo: Craig Butt

Earlier this month The Age’s data journalism team opened the doors to Melbourne’s data community in the form of a hackathon. The aim of the event was to explore the relationship between big data to drive a narrative in the form of data visualisation.

It was a fascinating experiment that saw programmers, data crunchers, journalists, graphic designers and open data activists come together to ask the question, at least in my mind, if data-driven journalism is art or science.

Having worked on several data-driven pieces for our masthead news sites over the last year, I know that I started with a fair amount of enthusiastic swagger that, given enough data and processing power, I could discover amazing stories. A data whisperer if you will.

It didn’t take long to knock the wind out of my sails. 

We live in a society where the desire for open data outpaces the ability (or willingness) of governments and businesses to provide it. Half the battle at times is just getting the data into usable format and then keeping it current.

Once you have a validated and repeatable process for ingesting the data, then comes the hunt for the story. Here, the data often becomes frustrating for what it hints at rather than clearly proves. You think you’ve found an interesting thread, and it starts raising more questions than answers.

At this point my journalist colleagues smile knowingly at me and pat my shoulder when they see this frustration. They have been there before.

Over time, my expectations have been lowered, but not my excitement at being given a big chunk of data to explore. Data driven journalism is both science and art - stories rarely just fall out automatically.

So it was at the Hackathon event last week. It was gratifying at least to me, that these teams went through these emotions. The initial story pitches were all very promising and by the closing bell the participants had that frantic look in their eyes that the analysis still proved inconclusive - the story remained elusive.

Each group had a seed of a great story - there were concrete questions raised by the data analysis. Many of the groups studied the AEC political donation returns and mashed it together with our historical news archives to find some interesting donation behaviours which bear some closer scrutiny.

The day ended with great solidarity. In just a few hours we created a little community of data hunters which combined different disciplines and I think the insights gained were welcomed by all.

The process

Fairfax Media made available to the participants a collection of different types of data. Weatherzone provided a large weather time series of readings from most of their data collection sensors from around the country. 

The second data set was advanced access to an experimental collection of 25 years of news text pre-marked with named entities (people, places, businesses, brands, etc).

The final data set was the freshly updated AEC political donations data as far back as 1998-1999.

Teams were organise to try to spread out the developer, journalism and design talent. Each team was given some time to have an initial look at the data and get a feel for what they thought would make good story seeds. Each group did a 30-second elevator pitch to an editorial panel to get some guidance and then a final presentation at the end of the day to the same panel.

The day was made possible by the assistance of and promotion by The Age, Melbourne University, the Australian National Data Service and the Open Knowledge Foundation. Additional data expertise was provided by the Computational Linguistics research group from Sydney University and data hosting was provided by Amazon.

You can read more about the day and the winners on the Data Point blog.

The author attended the event as a data mentor and also sat on the editorial panel. 

Would you like to participate in more events like this in Sydney, Melbourne and Brisbane? What data would you like to unleash on a community of data crunchers? Let us know in the comments.

This author is on Google+ 

Follow IT Pro on Twitter