Polling wizard conjures up perfect prediction
Nate Silver has been vindicated after his predictions came good two US presidential elections in a row. Photo: JD Lasica, Socialmedia.biz
The political emperors have no clothes, stripped bare by a big-data wizard named Nate Silver who showed dispassionate maths was more reliable than pundit intuition and cherry-picked polls.
Silver, 34, a statistician who previously predicted the career trajectories of baseball players, accurately tipped 49 out of 50 US states (with the 50th, Florida, highly likely to be accurate as well as Obama is ahead with 97 per cent of the votes counted) and most Senate contests.
This puts a check on the traditional pundits and the state of punditry in general. It makes me wonder if we have a changing of the guard.Clifford Young, managing director of polling at firm Ipsos
It's either better than or a repeat of his performance in the 2008 US presidential elections, when he accurately predicted 49 out of 50 states and all Senate races.
Nate Silver wins "best political blog" at the 16th Annual Webby Awards in May in New York City. Photo: Paul Zimmerman/Getty Images
As right-wing pundits attacked him and his “voodoo statistics” for failing to see that the election was on a knife edge – and in the case of some conservative wingnuts, for being openly gay and “effeminate” – Silver held his nerve and for the entire election cycle maintained that the data always pointed to an easy Obama victory.
Sales of his book, The Signal and the Noise: Why So Many Predictions Fail - But Some Don't, have rocketed 850 per cent on Amazon during the past 24 hours and it is now the second best-selling book on the site.
His Twitter following has ballooned to 374,954, and there's even a Chuck Norris-esque meme, with trending hashtag #natesilverfacts.
Several blogs and news sites dubbed him the real winner of the election. “Data is vindicated ... So who's an embarrassment to journalism now?” tweeted the Poynter Institute.
The day before the election, 20 per cent of visitors to the New York Times website went to Silver's blog, Five Thirty Eight.
Even after Obama's dismal first debate performance, Silver's probability of Obama winning never dipped below 61.1 per cent, rising to more than 90 per cent on election day.
Rather than focus on data from individual polls, Silver developed a complex algorithm that analysed hundreds of state and national polls, weighted the results based on various factors including past accuracy of the polling firms, and then added other data such as candidates' history and how much money they had raised. It simulated hundreds of thousands of elections, spitting out predictions as new data came in.
"Ultimately, what he's done is take a lot of the mysticism out of politics. This puts a check on the traditional pundits and the state of punditry in general. It makes me wonder if we have a changing of the guard," said Clifford Young, managing director of polling at firm Ipsos.
The Australian big data pioneer Anthony Goldbloom yesterday said big data techniques such as the ones Silver used were “starting to replace expert knowledge”.
On Thursday he reiterated those remarks but said Nate's perfect score was “less impressive than it seems at first blush”. “It turns out that for this election, a simple average of the polls also predicted all 50 states correctly,” he said.
The popular vote may have been a toss-up but the electoral college votes, according to the data, were never in Mitt Romney's favour, despite a laundry list of pundits and pollsters predicting a Romney win or missing key states.
Rasmussen Reports, for instance, was wrong on six of the nine swing-state polls and showed Romney winning the popular vote by one percentage point. The NBC News/Wall Street Journal/Marist College poll incorrectly predicted that Obama would win North Carolina, while the CBS/Quinnipiac University poll incorrectly showed Obama losing Colorado.
To be sure, several poll aggregators such as RealClearPolitics - and the betting markets on Intrade and Betfair - predicted an easy Obama victory, but precious few got all states correct, with Silver among the only analysts predicting Florida would be won by Obama, by a tiny margin.
The other big data geeks with perfect scores this election, who inspired far less controversy, included Stanford University's Simon Jackman and Drew Linzer from Votomatic.
As Silver said himself in the lead-up, he was destined to get “too much credit if the prediction is right and too much blame if it is wrong”.
Wired magazine declared it the “nerdiest election ever”: where 2008 was about hope and change, 2012 was about data and memes. Both campaigns used highly sophisticated data mining and analysis - including factors such as age, wealth, location, voting history, etc - to understand and target voters and donors.
Time magazine has published an exclusive look at how the big data-savvy Obama campaign was able to optimise its strategy to target voters who could be swayed and not waste time on those unlikely to vote blue.
In the lead-up to the election, Politico wondered whether Silver would be a “one-term celebrity”, MSNBC's Joe Scarborough called him a “joke” and he was even dismissed by New York Times colleague David Brooks.
He has been surprisingly gracious in victory, giving a plug to his book but refusing to have a dig at his critics and declining to respond to questions from Fairfax and others.
"He relies on his track record," said Colby Hall, founding editor of Medialite.com. The nerdy shtick works in Silver's favor. "If Jon Stewart and Stephen Colbert are falling over themselves to talk to you, you are in the hippest/coolest/most-insidery group."
But the Twittersphere was doling out more than its fair share of jokes and recriminations. “You know who won the election tonight? Nate Silver,” wrote MSNBC's Rachel Maddow.
For the Nate-haters, here’s the 538 prediction and actual results side by side twitter.com/cosentino/stat…— Michael Cosentino (@cosentino) November 7, 2012
Even the aforementioned Politico reporter who rounded up the negative commentary about Silver, Dylan Byers, tweeted: “Nate Silver nailed it.”
- with Reuters