Intel forecasts are a unique science
Amir Bagherpour already has a detailed set of charts predicting how everything will play out in the NAFTA negotiations, even though they don’t actually start for another few weeks.
He makes predictions for a living.
The U.S. intelligence community runs a prediction market where forecasters across government compete for prognosticative supremacy — it looks like a golf tournament leaderboard, only instead of birdies and bogeys, people are ranked by how correctly they call coup d’etats and counterinsurgencies.
Bagherpour was one of them. He was a State Department analyst under the Democrats and made predictions about things like Israeli-Palestinian peace, the Syrian conflict, Colombia’s negotiations with the FARC rebels, and the counter-ISIS campaign.
His predictions are often bang on. He believed Donald Trump might win the presidency. He wrote a paper five years ago that predicted Bashar Assad would cling to power, with Syria’s conflict spiralling into a stalemate defined by religion. Sometimes they miss the mark: he gave Brexit a onethird chance of success.
The administration he served took an active interest in the science of forecasting: “(Barack) Obama would ask, ‘Where’s the prediction market on this (topic)?’” says Bagherpour, who now runs a consultancy, Global Impact Strategies.
The U.S. intelligence community has created more than a half-dozen forecasting programs over the last few years through its research unit, the Intelligence Advanced Research Projects Activity (IARPA), modelled after the older Defense Advanced Research Projects Agency (DARPA) that helped create the internet.
One example is an ongoing tournament between hybrid teams combining humans and machines. It’s based on evidence that the best forecasting comes from a combination of computer algorithm and human guidance.
“We love the concept of forecasting tournaments,” said Seth Goldstein, who is running IARPA’s human-machine Hybrid Forecasting Competition. He’s limited in what he can say about the tournment, but offers one example of how it works.
“(We might ask), ‘Will Leader A in Country B be removed from power, by Date C?’ That would be the type of question ... We see what techniques work, and what techniques don’t work...
“These tournaments (give us) a pretty good indication.”
Participants come from all walks of life, in academia and industry, and receive a stipend for taking part. But there are no rewards for accurate predictions. That’s the lingering legacy of an old controversy, which forced a project to be shelved and the Pentagon boss running it to resign.
The source of controversy: a terrorism futures market. Created after the 9/11 attacks, participants were allowed to place bets on the occurrence of future terrorist acts — which critics viewed as tasteless, at best, and as a dangerous perverse incentive at worst.
The program was swiftly cancelled in 2003.
The initiative was reborn with a new generation of projects years later. And Canadians played a major role in the resurrection.
The team that dominated the first IARPA tournament was cocreated by Philip Tetlock, a researcher, author, and University of Pennsylvania professor who was born in Toronto, and raised in Winnipeg and Vancouver.
His team beat a control group by a whopping 60 per cent and 78 per cent in the competition’s first two years starting in 2011. It was so lopsided they ended the competition, and Tetlock’s team continued alone.
The U.S. government has just released the data collected from his team to help future researchers.
Some secrets to successful forecasting are quite simple, Tetlock says. He includes a socalled Ten Commandments in his book, “Super-forecasting: The Art and Science of Prediction,” coauthored with Canadian writer and public servant Dan Gardner.
One trick: doubt yourself. Assume your prediction is wrong, ask why, and incorporate that doubt factor into your assessment. Another is to tackle a problem in pieces — break the question into bite-sized chunks.
For example, Tetlock’s book cites a love-starved London forecaster who wants to determine his number of potential mates. He takes the local population figure, divides it by two for gender, isolates an age range, the likely singles population, the universityeducated percentage, and finally the percentage he will likely attract and be attracted to.
His conclusion: he has 26 potential mates in London.
Tetlock says great forecasters use this approach. These are the people who score highest on the zero-to-two Brier scale, the standard unit for measuring predictive success.
“I’m not talking about people who have Nostradamus clairvoyance properties,” Tetlock said in an interview.
“We’re talking about people who are better at assigning realistic odds to everything. Does that mean they’re going to see everything — that whenever history hits a sharp corner they’re going to be able to see around the corner? Absolutely not. There are limits on foresight...
“It helps to be smart. It helps to be well-informed.”