Philip E. Tetlock

How to predict
Co-author Superforecasting: The Art and Science of Prediction

How to create superforecasters — those who see clearly into the future

Add to Shortlist GoodJudgement.com

Biography

Philip Tetlock is the Annenberg University Professor at the University of Pennsylvania, with appointments in Wharton, psychology and political science. He is co-leader of the Good Judgement Project, a multi-year forecasting study, the author of Expert Political Judgement and co-author of Superforecasting.

In Superforecasting, Tetlock offers a masterwork on prediction, drawing on decades of research and the results of massive, government-funded forecasting tournament. The Good Judgment Project involves tens of thousands of ordinary — including a Brooklyn filmmaker, a retired pipe installer, and a former ballroom dancer — who set out to forecast global events. Some of the volunteers have turned out astonishingly good. They've beaten other benchmarks, competitors, and prediction marks. They've even beaten the collective judgement of intelligence analysts with access to classified information. They are “super forecasters.”

"The techniques and habits of mind set out in this book are a gift to anyone who has to think about what the future might bring. In other words, to everyone." (The Economist)

"A terrific piece of work that deserves to be widely read... Highly recommended." (Independent)

"This marvelous book tells an exciting story of ordinary people beating experts in a very serious game. It is also a manual for thinking clearly in an uncertain world. Read it." (Daniel Kahneman)

"Full of excellent advice – it is the best thing I have read on predictions... Superforecasting is an indispensable guide to this indispensable activity." (The Times)

"Philip Tetlock has transformed the science of prediction." (Spectator)

"Superforecasting is a fascinating book." (Daily Mail)

"Superforecasting is a very good book. In fact it is essential reading." (Management Today)

"The best way to know if an idea is right is to see if it predicts the future. But which ideas, which methods, which people have a track record of non-obvious predictions vindicated by the course of events? The answers will surprise you, and they have radical implications for politics, policy, journalism, education, and even epistemology – how we can best gain knowledge about the world. The casual style of Superforecasting belies the profundity of its message." (Steven Pinker)

"There isn't a social scientist in the world I admire more than Phil Tetlock." (Tim Harford)

"Superforecasting is a rare book that will make you smarter and wiser. One of the giants of behavioral science reveals how to improve at predicting the future." (Adam Grant)

Books

Superforecasting

The Art and Science of Prediction

Philip E. Tetlock and Dan Gardner

Everyone would benefit from seeing further into the future, whether buying stocks, crafting policy, launching a new product, or simply planning the week’s meals. Unfortunately, people tend to be terrible forecasters. As Wharton professor Philip Tetlock showed in a landmark 2005 study, even experts’ predictions are only slightly better than chance. However, an important and underreported conclusion of that study was that some experts do have real foresight, and Tetlock has spent the past decade trying to figure out why. What makes some people so good? And can this talent be taught?

In Superforecasting, Tetlock and coauthor Dan Gardner offer a masterwork on prediction, drawing on decades of research and the results of a massive, government-funded forecasting tournament. The Good Judgment Project involves tens of thousands of ordinary people — including a Brooklyn filmmaker, a retired pipe installer, and a former ballroom dancer — who set out to forecast global events. Some of the volunteers have turned out to be astonishingly good. They’ve beaten other benchmarks, competitors, and prediction markets. They’ve even beaten the collective judgment of intelligence analysts with access to classified information. They are "superforecasters."

In this groundbreaking and accessible book, Tetlock and Gardner show us how we can learn from this elite group. Weaving together stories of forecasting successes (the raid on Osama bin Laden’s compound) and failures (the Bay of Pigs) and interviews with a range of high-level decision makers, from David Petraeus to Robert Rubin, they show that good forecasting doesn’t require powerful computers or arcane methods. It involves gathering evidence from a variety of sources, thinking probabilistically, working in teams, keeping score, and being willing to admit error and change course. Superforecasting offers the first demonstrably effective way to improve our ability to predict the future — whether in business, finance, politics, international affairs, or daily life — and is destined to become a modern classic.

Random House Books (7 April 2016)
Random House Books (24 Sept. 2015)

Reviews

Is Forecasting A General Skill set? — Fisher Investments MarektMinder
'Mindware' and 'Superforecasting'The New York Times Sunday Book Review SuperforecastingKirkus Reviews

Praise

“I think Philip Tetlock’s ‘Superforecasting: The Art and Science of Prediction,’ co-written with the journalist Dan Gardner, is the most important book on decision making since Daniel Kahneman’s ‘Thinking, Fast and Slow.'”
— Jason Zweig, Wall Street Journal

“Mr Tetlock and his collaborators were able to extract some common personality traits. Superforecasters are clever, on average, but by no means geniuses. More important than sheer intelligence was mental attitude… the techniques and habits of mind set out in this book are a gift to anyone who has to think about what the future might bring. In other words, to everyone.”
The Economist

"Just as modern medicine began when a farsighted few began to collect data and keep track of outcomes, to trust objective “scoring” over their own intuitions, it’s time now for similar demands to be made of the experts who lead public opinion. It’s time for evidence-based forecasting."
— Sam Winter-Levy and Jacob Trefethen, Washington Post

“Tetlock and Gardner believe anyone can improve their forecasting ability by learning from the way they work. If that’s true, people in business and finance who make an effort to do so have a lot to gain – and those who don’t, much to lose.”
Financial Post

Superforecasting is a very good book. In fact it is essential reading – which I have never said in any of my previous MT reviews… It should be on every manager’s and investor’s reading list around the topics du jour of decision-making, prediction and behavioural economics”
— Andrew Wileman, Management Today

“[O]ne of Tetlock’s key points is that these aren’t innate skills: they can be both taught and learned… Tetlock’s ‘Ten Commandments For Aspiring Superforecasters’ should probably have a place of honor in most business meeting rooms.”
— Frank David, Forbes

“I’ve been hard on social science, even suggesting that ‘social science’ is an oxymoron. I noted, however, that social science has enormous potential, especially when it combines ‘rigorous empiricism with a resistance to absolute answers.’ The work of Philip Tetlock possesses these qualities”
— John Horgan, Scientific American

“We ought to expect that everybody who aspires to high office is trained to understand why they are so likely to make mistakes forecasting complex events… Politics is harder than physics but Tetlock has shown that it doesn’t have to be like astrology.”
— Dominic Cummings, The Spectator

“Philip E. Tetlock, a professor at the University of Pennsylvania and co-leader of a multiyear online forecasting study, the Good Judgment Project, and Dan Gardner, a journalist, teamed up to produce one of the best books I’ve read this year.”
— Brenda Jubin, Seeking Alpha

“The key to becoming a better forecaster, if not a super one, according to Tetlock is the same as any other endeavor: Practice, practice, practice.”
— Gregg Greenberg, The Street

“Philip Tetlock is the world expert on a vital subject. Superforecasting is the wonderful story of how he and his research team got ordinary people to beat experts in a very serious game. It is also a manual for thinking clearly in an uncertain world. Read it.”
— Daniel Kahneman, winner of the Nobel Prize and author of Thinking, Fast and Slow

“The best way to know if an idea is right is to see if it predicts the future. But which ideas, which methods, which people have a track record of non-obvious predictions vindicated by the course of events? The answers will surprise you, and they have radical implications for politics, policy, journalism, education, and even epistemology — how we can best gain knowledge about the world. The casual style of Superforecasting belies the profundity of its message.”
— Steven Pinker, Johnstone Professor of Psychology, Harvard University, and author of The Better Angels of Our Nature

“In this accessible and lively book, Tetlock and Gardner recognize the centrality of probabilistic thinking to sound forecasting. Whether you are a policymaker or anyone else who wants to approach decisions with great rigor, Superforecasting will serve as a highly useful guide.”
— Robert E. Rubin, Former U.S. Treasury Secretary

“How well can we predict the future, really? There is no better way to answer that question than to read this book. You will come away disillusioned about the ability of experts, but also enlightened about how the best forecasters do it — and maybe even hopeful about your own prospects.”
— Tyler Cowen, Director of the George Mason University Mercatus Center and author of Average Is Over

“For thousands of years, people have listened to those who foretold the future with confidence and little accountability. In this book, Tetlock and Gardner free us from our foolishness. Full of great stories and simple statistics, Superforecasting gives us a new way of thinking about the complexity of the world, the limitations of our minds, and why some people can consistently outpredict a dart-throwing chimp. Tetlock’s research has the potential to revolutionize foreign policy, economic policy, and your own day-to-day decisions.”
— Jonathan Haidt, New York University Stern School of Business, and author of The Righteous Mind

Superforecasting is a rare book that will make you smarter and wiser. One of the giants of behavioral science reveals how to improve at predicting the future.”
— Adam Grant, New York Times bestselling author of Give and Take

“Good judgment and good forecasting are rare, but they turn out to be made of teachable skills. By forcing forecasters to compete, Tetlock discovered what the skills are and how they work, and this book teaches the ability to any interested reader.”
— Stewart Brand, President, The Long Now Foundation

“Philip Tetlock is renowned for demonstrating that most experts are no better than ‘dart-throwing monkeys’ at predicting elections, wars, economic collapses and other events. In his brilliant new book, Tetlock offers a much more hopeful message, based once again on his own ground-breaking research. He shows that certain people can forecast events with accuracy much better than chance — and so, perhaps, can the rest of us, if we emulate the critical thinking of these ‘superforecasters.’ The self-empowerment genre doesn’t get any smarter and more sophisticated than this.”
— John Horgan, Director, Center for Science Writings, Stevens Institute of Technology

Superforecasting is the rare book that is both scholarly and engaging. The lessons are scientific, compelling, and enormously practical. Anyone who is in the forecasting business — and that’s all of us — should drop what they are doing and read it.”
— Michael J. Mauboussin, Head of Global Financial Strategies, Credit Suisse

“There isn’t a social scientist in the world I admire more than Phil Tetlock.” — Tim Harford, author of The Undercover Economist

“From the Oracle of Delphi to medieval astrologers to modern overconfident experts, forecasters have been either deluded or fraudulent. For the first time, Superforecasting reveals the secret of making honest, reliable, effective, useful judgments about the future.”
— Aaron Brown, Chief Risk Officer of AQR Capital Management and author of The Poker Face of Wall Street

“Socrates had the insight in ‘know thyself,’ Kahneman delivered the science in Thinking, Fast and Slow, and now Tetlock has something we can all apply in Superforecasting.”
— Juan Luis Perez, Global Head of UBS Group Research

"In this captivating book, Tetlock argues that success is all about the approach: foresight is not a gift but rather a product of a particular way of thinking... In each chapter, the author augments his research with compelling interviews, anecdotes, and historical context, using accessible real-world examples to frame what could otherwise be dense subject matter. His writing is so engaging and his argument so tantalizing, readers will quickly be drawn into the challenge — in the appendix, the author provides a concise training manual to do just that. A must-read field guide for the intellectually curious."
— Kirkus Reviews, starred

Expert Political Judgment

How Good Is It? How Can We Know?

Philip E. Tetlock

The intelligence failures surrounding the invasion of Iraq dramatically illustrate the necessity of developing standards for evaluating expert opinion. This book fills that need. Here, Philip E. Tetlock explores what constitutes good judgment in predicting future events, and looks at why experts are often wrong in their forecasts.

Tetlock first discusses arguments about whether the world is too complex for people to find the tools to understand political phenomena, let alone predict the future. He evaluates predictions from experts in different fields, comparing them to predictions by well-informed laity or those based on simple extrapolation from current trends. He goes on to analyze which styles of thinking are more successful in forecasting. Classifying thinking styles using Isaiah Berlin's prototypes of the fox and the hedgehog, Tetlock contends that the fox — the thinker who knows many little things, draws from an eclectic array of traditions, and is better able to improvise in response to changing events — is more successful in predicting the future than the hedgehog, who knows one big thing, toils devotedly within one tradition, and imposes formulaic solutions on ill-defined problems. He notes a perversely inverse relationship between the best scientific indicators of good judgement and the qualities that the media most prizes in pundits — the single-minded determination required to prevail in ideological combat.

Clearly written and impeccably researched, the book fills a huge void in the literature on evaluating expert opinion. It will appeal across many academic disciplines as well as to corporations seeking to develop standards for judging expert decision-making.

Princeton University Press; New Ed edition (August 20, 2006)

Awards

Winner of the 2006 Grawemeyer Award for Ideas Improving World Order

Winner of the 2006 Woodrow Wilson Foundation Award, American Political Science Association

Winner of the 2006 Woodrow Wilson Foundation Award, American Political Science Association

Winner of the 2006 Robert E. Lane Award, Political Psychology Section of the American Political Science Association

Praise

"It is the somewhat gratifying lesson of Philip Tetlock's new book . . . that people who make prediction their business — people who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtables — are no better than the rest of us. When they're wrong, they're rarely held accountable, and they rarely admit it, either. . . . It would be nice if there were fewer partisans on television disguised as "analysts" and "experts". . . . But the best lesson of Tetlock's book may be the one that he seems most reluctant to draw: Think for yourself."
— Louis Menand, The New Yorker

"The definitive work on this question. . . . Tetlock systematically collected a vast number of individual forecasts about political and economic events, made by recognised experts over a period of more than 20 years. He showed that these forecasts were not very much better than making predictions by chance, and also that experts performed only slightly better than the average person who was casually informed about the subject in hand."
— Gavyn Davies, Financial Times

"Before anyone turns an ear to the panels of pundits, they might do well to obtain a copy of Phillip Tetlock's new book Expert Political Judgment: How Good Is It? How Can We Know? The Berkeley psychiatrist has apparently made a 20-year study of predictions by the sorts who appear as experts on TV and get quoted in newspapers and found that they are no better than the rest of us at prognostication."
— Jim Coyle, Toronto Star

"Tetlock uses science and policy to brilliantly explore what constitutes good judgment in predicting future events and to examine why experts are often wrong in their forecasts."
Choice

"[This] book . . . Marshals powerful evidence to make [its] case. Expert Political Judgment . . . Summarizes the results of a truly amazing research project. . . . The question that screams out from the data is why the world keeps believing that "experts" exist at all."
— Geoffrey Colvin, Fortune

"Philip Tetlock has just produced a study which suggests we should view expertise in political forecasting--by academics or intelligence analysts, independent pundits, journalists or institutional specialists — with the same skepticism that the well-informed now apply to stockmarket forecasting. . . . It is the scientific spirit with which he tackled his project that is the most notable thing about his book, but the findings of his inquiry are important and, for both reasons, everyone seriously concerned with forecasting, political risk, strategic analysis and public policy debate would do well to read the book."
— Paul Monk, Australian Financial Review

"Phillip E. Tetlock does a remarkable job . . . applying the high-end statistical and methodological tools of social science to the alchemistic world of the political prognosticator. The result is a fascinating blend of science and storytelling, in the the best sense of both words."
— William D. Crano, PsysCRITIQUES

"Mr. Tetlock's analysis is about political judgment but equally relevant to economic and commercial assessments."
— John Kay, Financial Times

"Why do most political experts prove to be wrong most of time? For an answer, you might want to browse through a very fascinating study by Philip Tetlock . . . who in Expert Political Judgment contends that there is no direct correlation between the intelligence and knowledge of the political expert and the quality of his or her forecasts. If you want to know whether this or that pundit is making a correct prediction, don't ask yourself what he or she is thinking — but how he or she is thinking."
— Leon Hadar, Business Times

Videos

Predicting the future | AEI

Panel Discussion: Predicting Elections | Columbia University

Philip begins about 29:10 minutes.

Feedback

A financial company:
Phil's presentation was outstanding, and the feedback [we] received was extremely positive.

Articles

— The Atlantic
— The Philadelphia Inquirer
— The Globe and Mail
— The Globe and Mail
— Financial Times
— New York Magazine
— Harvard Business Review
— ValueWalk
— The Telegraph
— InTheBlack
— The Washington Post
— Corporate Counsel
— Financial Times
— Edge
— Daily Reckoning UK
— The Wall Street Journal
— The Wall Street Journal
— The Daily Pennsylvanian
— The New York Times
— The New York Times
— The Economist
— The New York Times
— Edge