Why do pollsters and pundits keep getting it wrong?

Trump Wins: Another Surprise?

By Nigel Jacklin

FOR the past week I’ve been seeing polls, predictions and pronouncements on the US presidential election.

Many of these have had an air of certainty which belies the data and belittles the democratic process. Yet this morning, here in the UK, we wake up to hear that the US electorate has chose Trump to be their president again. Should we be surprised?

Some time ago I did some work for the FT comparing the quality of their product with a key competitor (the Wall Street Journal). I selected a number of key news events and evaluated how the two newspapers covered them from the perspective of a time-poor senior business decision maker.

One event I looked at was the crash of the Chinese Stock market. Here the FT quickly told me it ‘was not an indicator of the wider Chinese economy.’ If I’d been meeting a Chinese investor I could relax.

I also looked at the nomination of Trump for the 2016 Presidential election. Here the FT quickly told me ‘it seems likely Trump will win the Republican nomination’. They ran a single, conclusive article. On the same day the Wall Street Journal ran eight articles which mostly opined that Trump was a terrible man and he’d be a bad choice for the USA. From the point of view of an ‘executive in a hurry’ the FT won the day.

The reaction to Trump’s election in 2016 was a flurry of allegations. Guardian journalist Caroline Cadwallah wrote a whole book citing the impact of social media and highlighted the possible role of a British firm, Cambridge Analytica. The idea that something was wrong was ‘confirmed’ when the UK voted to leave the EU.

Another theory was that the Russians were behind all this. I took a look at it and sought a third result to test out this hypothesis. The 2017 UK general election provided one: Jeremy Corbyn did much better than expected (but not well enough to win). I really couldn’t fit Corbyn’s popularity into the Cambridge Analytica or the ‘Russian’ hypotheses.

I compared Cadwallah’s book with one by her colleague citing Russia. Cross-referencing the indexes of the two, one was full of references to Facebook, the other to Russia, but Facebook wasn’t a feature of the Russia book and Russia wasn’t a feature of Cadwallah’s. How could two people produce such different analyses of the same thing? Were they, like the Wall Street Journal, simply obsessed with their own ideas and opinions?

So when we came to this years’ US election, I was disappointed to see both The Economist and the FT endorse Kamala Harris. The Economist’s election poll tracker ended up predicting a victory for Harris, but did present the results as a narrow chance of winning.

This really showed that the election was a close call. So why not lead on this? Why make out there is certainty, when it clearly all depended on who turned up and how they voted.

Are pollsters to blame?

Back when I first looked at this, the Royal Statistical Society convened a meeting to discuss why the UK election pollsters kept getting it wrong. At the time I was working as a consultant to the UK newspapers on a different project renewing the service which measures readership. The contractor also ran polls and the CEO was at the meeting. He was also the senior point person on the readership contract. I had previously told the newspapers that the old survey was not fit for purpose and when the new proposals came out that they were flawed.

We had changed the survey design in line with my recommendations and then needed to test integrating two datasets. Early on I said ‘it’s not going to work; we need a plan B’. The contractor’s response was that ‘Plan B is to work harder on Plan A’. Thirteen months later Plan A was abandoned and we went back to the drawing board. I had been right.

At the Royal Statistical Society meeting the same ‘confident’ manner was taken up by the pollsters. They assured us they could ‘learn from the lessons of the past’ and introduce appropriate adjustments into their models. I disagreed. It was patently obvious that each election had presented a different set of challenges and the pollsters were clearly not up to tackling them. If the future is not the same as the past, adjusting based on the past will just make things worse.

At the meeting I approached my client CEO. He turned his back on me, more interested in talking to people who would confirm he was right. Why do pollsters and pundits keep getting it wrong?

Democracy really is a wonderful thing. The mechanics of an election, from the campaign period through to the final count really can result in the people choosing their elected representative (I realise it may not feel like that here in the UK right now). The results actually do depend on who turns up and how they vote (half of voting age people did not vote in this years UK general election). Polls cannot accurately reflect or predict that. They can say ‘the election looks close’ and that’s really what they should have said this time in the US.

So why did the FT, The Economist and others come out with an endorsement and in some cases a prediction? Is it arrogance or ignorance?

When I was working for the FT back in the early days of the internet, whilst the journalist’s name would be put to an article, they were generally writing for ‘the paper’. As FT.com grew in importance the journalists were told they needed to have their picture taken and this would appear alongside their content. Some were uncomfortable with this. It was a significant step in the growth of ‘celebrity’ journalism. Named ‘personality’ contributors also grew in importance.

So in recent days, as well as the FT and The Economist, I’ve had the benefit of William Hague and George Monbiot’s opinions on who I should vote for, (notwithstanding the fact that I can’t vote in the US election) and we’ve been told our own Government has sent a team of people to support Harris. I think this is beyond misguided. The arrogance of the British media and political class shows a lack of respect for the voice of the American people. It’s their choice, their election, let them decide. We then work with whoever they choose.

This ‘false certainty’ was at its height in 2016 when the EU failed to prepare for a Trump victory. That arrogance was one reason many people voted to leave the EU, a choice which still seems to be a surprise to some.

So now we will now have a new US President again. I’m sure Nigel Farage will be smiling.

If Trump can end the war in Ukraine, all of us Brits and Europeans should smile too.

• Nigel Jacklin is a market researcher and statistician. He stood as an Independent candidate in the UK General Election. He is @TheGoodStatsMan on X.