Superforecasting: The Art & Science of Prediction

Philip E. Tetlock & Dan Gardner “Superforecasting: The Art & Science of Prediction” Crown Publishers, 2015, 285 pp.

This is a fantastic book and a must-read for anyone interested in making better decisions and predictions.  A quick simplification of the premise is that the closest most people come to thinking probabilistic is to divide a decision up into yes, no, and maybe.  This book throws out yes and no (except when you see a shadow in the grass and it might be a lion, or if you are a TV pundit), which only leaves “maybe.”  “Maybe”, then demands, if you’re a superforecaster, analysis and reasoned assignment of a percentage from 0.00.. to 99.99… along with a commitment to keep studying the problem and adjusting your forecast accordingly.

Another characteristic of superforecasters is their welcoming attitude to a challenge.  For instance, when asked who is likely to be the next elected President of Ghana most people, including me, would just not care.  A superforecaster would look it as a challenge and opportunity to learn something about something new.  Also, the strongest predictor of rising into the ranks of superforecasters is perpetual beta, the degree to which one is committed to belief updating and self-improvement.  It is roughly three times as powerful a predictor as its closest rival, intelligence.  For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded.

This book explains author Tetlock’s organization of volunteer teams to compete and make predictions about items our intelligence analysts normally investigate.  Examples include “Will the president of Tunisia flee to exile?” “Will the euro fall below $1.20 in the next twelve months?”  Tetlock then studied the resulting forecasts from all the volunteers and called the top 2% “superforecasters.”

My Notes:

P. 8:  In 1972 the American meteorologist Edward Lorenz wrote a paper with an arresting title: “Predictability: does the Flap of a Butterfly’s Wings in Brazil Set Off a Tornado in Texas?”  Lorenz had discovered that tiny data entry variations in computer simulations of weather patterns—like replacing 0.506127 with 0.506—could produce dramatically different long-term forecasts.  It was an insight that would inspire “chaos theory”: in nonlinear systems like the atmosphere, even small changes in initial conditions can mushroom to enormous proportions. 

Pg. 18:  Foresight isn’t a mysterious gift bestowed at birth.  It is the product of particular ways of thinking, of gathering information, of updating beliefs.  These habits of thought can be learned and cultivated by any intelligent, thoughtful, determined person. 

Pg. 19:  The medical profession finally accepted that their experience and perceptions were not reliable means of determining whether a treatment works and turned to scientific testing.  Medicine then finally started to make rapid advances.  The same revolution needs to happen in forecasting.

Pg. 28:  The cure for scurvy came close in 1747 when a British ship’s doctor took twelve sailors suffering from scurvy, divided them into pairs, and gave each pair a different treatment: vinegar, cider, sulfuric acid, seawater, a bark paste, and citrus fruit.  The two sailors given the citrus recovered quickly.  But the doctor had no full understanding of what he was doing and failed to make sense of his own experiment and was left unconvinced of the exceptional benefits of lemons and limes.  Not until the twentieth century did the idea of randomized trial experiments, careful measurement, and statistical power take hold.  (Vitamin C is the answer)

Pg. 33:  How we think is now commonly described by using a model that partitions our mental universe into two domains.  System 2 is the familiar realm of conscious thought.  It consists of everything we choose to focus on.  By contrast, System 1 is largely a stranger to us.  It is the realm of automatic perceptual and cognitive operations—like those you use when reading to transform the print on a page into a meaningful sentence or to hold the book while reaching for a glass and taking a sip. We have no awareness of these rapid-fire processes but we could not function without them.  We would shut down.

The numbering of the two systems is not arbitrary.  System 1 comes first.  It is fast and constantly running in the background.  If a question is asked and you instantly know the answer, it sprang from System 1.  System 2 is charged with interrogating that answer.  Does it stand up to scrutiny?  Is it backed by evidence?  This process takes time and effort, which is why the standard routine in decision making is this: first System 1 delivers an answer and only then can System 2 get involved, starting with an examination of what System 1 decided.  For instance, answer this: A bat and ball together cost $1.10.  The bat costs a dollar more than the ball.  How much does the ball cost?  (The answer is NOT $0.10).

Pg. 118:  A superforecaster when asked to estimate something starts with the “outside” view.  For instance, how likely a particular family is to own a pet is determined first by finding the outside view likelihood.  We google it and find that 62% of American households have pets.  Then move to the specific characteristics of a specific family and accordingly move that estimate up or down.

Pg. 137:  It wasn’t until publication of Jakob Beroulli’s Ars Conjectandi in 1713 that the best minds started to think seriously about probability.  One of twentieth-century science’s great accomplishments has been to show that uncertainty is an ineradicable element of reality. 

Pg. 153:  Forecasts aren’t like lottery tickets that you buy and file away until the big draw.  They are judgments that are based on available information and that should be updated in light of changing information.

Pg. 160:  Between February and August 1942 every man, woman, and child of Japanese descent in California (112,000 people) were shipped to camps and watched by armed guards.  Two-thirds of these Japanese had been born in the United States.  (There was no sabotage in the ten weeks before the internment, or in the rest of 1942.  Or in 1943. 

Pg. 174:  People have either a growth mindset or a fixed mindset.  The growth mindset people believe that your abilities are largely the product of effort—that you can “grow” to the extent that you are willing to work hard and learn.  The fixed mindset people believe that we are who we are, and abilities can only be revealed, not created and developed.  People with the fixed mindset say things like “I’m bad at math” and so they are.

Pg. 192:  The strongest predictor of rising into the ranks of superforecasters is perpetual beta, the degree to which one is committed to belief updating and self-improvement.  It is roughly three times as powerful a predictor as its closest rival, intelligence.  (Also: teams of superforecasters on average are 23% more accurate than individuals.  P.201)

Pg. 245:  Kahneman and other pioneers of modern psychology have revealed that our minds crave certainty and when they don’t find it, they impose it.  In forecasting, hindsight bias is the cardinal sin.  Recall how experts stunned by the Gorbachev surprise, quickly became convinced it was perfectly explicable, even predictable, although they hadn’t predicted it.

Pg. 265:  In December 2014 Thomas Friedman wrote about the ramification of the plunging price of oil.  “The last time the world witnessed such a steep and sustained drop in oil prices—from 1986 to 1999—it had some profound consequences for oil-dependent states and those who depended on their largesse.  The Soviet empire collapsed; Iran elected a reformist president; Iraq invaded Kuwait; and Yasir Arafat, having lost his Soviet backer and Arab bankers, recognized Israel—to name but a few.”  Friedman’s conclusion?  “If today’s falloff in oil prices is sustained, we’ll also be in for a lot of surprises”—particularly in the petro-states of Venezuela, Iran, and Russia.

Friedman in this writing presents us with a vague warning of unspecified surprises in unspecified time frames (with the unstated implication that oil prices caused these items).  As a forecast, that’s not terribly helpful.  This sort of thing is why some people see Friedman as a particularly successful and slippery pundit who has mastered the art of appearing to go out on a limb without ever venturing out.  But the same column can be read less as a forecast than an attempt to draw the attention of forecasters to something they should be thinking about.  In other words, it is a question, not an answer.

To learn more about superforecasting, go to:  www.goodjudgment.com