Why forecasting is harder than merely saying: “I knew it”
Iam not one of those clever people who claims to have seen the 2008 financial crisis coming, but 10 years ago I could see the fallout was going to be bad. The depth of the recession and the long-lasting hit to productivity came as no surprise. I knew it would happen.
Or did I? This is the story I tell myself, but I do not really know. I did not keep a diary and so must rely on memory — which, it turns out, is not reliable.
In 1972, psychologists Baruch Fischhoff and Ruth Beyth conducted a survey in which they asked for predictions about Richard Nixon’s imminent visit to China and Russia. How likely was it Nixon and Mao Zedong would meet? What chance was there the US would grant diplomatic recognition to China?
Fischhoff and Beyth wanted to know how people would remember their forecasts. Since their subjects had taken the unusual step of writing down a probability for each of 15 outcomes, one might have hoped for accuracy. But, no — the subjects flattered themselves hopelessly. The Fischhoff-Beyth paper was titled “I knew it would happen”. This is a reminder of what a difficult task we face when we try to make big-picture macroeconomic and geopolitical forecasts.
To start with, the world is complicated, which makes predictions challenging. For many subjects that interest us, there is a delay between the forecast and the outcome, and this delayed feedback makes it harder to learn from our successes and failures. Even worse, as Fischhoff and Beyth discovered, we systematically misremember what we once believed.
Small wonder that forecasters turn to computers for help. We have also known since work in the 1950s by the late psychologist, Paul Meehl, that simple statistical rules often outperform expert intuition. Meehl’s initial work focused on clinical cases — for example, faced with a patient suffering chest pains, could a two- or three-point checklist beat the judgment of an expert doctor? The experts did not fare well.
However, Meehl’s rules, like more modern machine-learning systems, require data to work. It is all very well for Amazon to forecast what effect a price drop may have on demand for a book — and some of the most successful hedge funds use algorithmically driven strategies — but trying to forecast the chance of Italy leaving the eurozone, or President Donald Trump’s impeachment, is not as simple. Machines are no better than we are. And they may be worse.
Much of what we know about forecasting, we know from the research of psychologist Philip Tetlock.
In the 1980s, Tetlock began to build on the Fischhoff-Beyth research by soliciting specific and often long-term forecasts from a variety of forecasters — initially hundreds. The results, described in his book Expert Political Judgement, were not encouraging. Yet his idea of evaluating large numbers of forecasters over a long time has blossomed, and some successful forecasters have emerged.
The latest step in this research is a Hybrid Forecasting Tournament sponsored by the US Intelligence Advanced Research Projects Activity, designed to explore ways in which humans and machinelearning systems can work together to produce better forecasts. We await the results.
If the computers do produce some insight, it may be because they can tap into data we could hardly have imagined using before. Satellite imaging can now track the growth of crops or the stockpiling of commodities such as oil.
Computers can guess at general human sentiment by analysing web searches for terms such as “job seekers allowance”, mentions of “recession” in news stories, and positive emotions in tweets.
In 1991, psychologist Harold Zullow published research suggesting that the emotional content of songs in the Billboard Hot 100 chart could predict recessions. Hits containing “pessimistic rumination” (“I heard it through the grapevine/Not much longer would you be mine”) tended to predict an economic downturn.
His successor is economist Hisam Sabouni, who reckons that a computer-aided analysis of Spotify streaming gives him an edge in forecasting stock market movements and consumer sentiment.
Will any of this prove useful for forecasting significant economic and political events? Perhaps. But for now, here is an easy way to use a computer to help you forecast: open up a spreadsheet, note down what you believe today, and regularly revisit and reflect.
The simplest forecasting tip of all is to keep score.
DELAYED FEEDBACK MAKES IT HARDER TO LEARN FROM OUR SUCCESSES AND FAILURES