Expert Prediction: hard and soft

Jason Zweig writes about Philip Tetlock's Good Judgement Project below. See also Expert Predictions, Perils of Prediction, and this podcast talk by Tetlock.

A quick summary: good amateurs (i.e., smart people who think probabilistically and are well read) typically perform as well as or better than area experts (e.g., PhDs in Social Science, History, Government; MBAs) when it comes to predicting real world outcomes. The marginal returns (in predictive power) to special "expertise" in soft subjects are small. (Most of the returns are in the form of credentialing or signaling ;-)

WSJ: ... I think Philip Tetlock’s “Superforecasting: The Art and Science of Prediction,” co-written with the journalist Dan Gardner, is the most important book on decision making since Daniel Kahneman’s “Thinking, Fast and Slow.” (I helped write and edit the Kahneman book but receive no royalties from it.) Prof. Kahneman agrees. “It’s a manual to systematic thinking in the real world,” he told me. “This book shows that under the right conditions regular people are capable of improving their judgment enough to beat the professionals at their own game.”

The book is so powerful because Prof. Tetlock, a psychologist and professor of management at the University of Pennsylvania’s Wharton School, has a remarkable trove of data. He has just concluded the first stage of what he calls the Good Judgment Project, which pitted some 20,000 amateur forecasters against some of the most knowledgeable experts in the world.

The amateurs won — hands down. Their forecasts were more accurate more often, and the confidence they had in their forecasts — as measured by the odds they set on being right — was more accurately tuned.

The top 2%, whom Prof. Tetlock dubs “superforecasters,” have above-average — but rarely genius-level — intelligence. Many are mathematicians, scientists or software engineers; but among the others are a pharmacist, a Pilates instructor, a caseworker for the Pennsylvania state welfare department and a Canadian underwater-hockey coach.

The forecasters competed online against four other teams and against government intelligence experts to answer nearly 500 questions over the course of four years: Will the president of Tunisia go into exile in the next month? Will the gold price exceed $1,850 on Sept. 30, 2011? Will OPEC agree to cut its oil output at or before its November 2014 meeting?

It turned out that, after rigorous statistical controls, the elite amateurs were on average about 30% more accurate than the experts with access to classified information. What’s more, the full pool of amateurs also outperformed the experts. ...

In technical subjects, such as chemistry or physics or mathematics, experts vastly outperform lay people even on questions related to everyday natural phenomena (let alone specialized topics). See, e.g., examples in Thinking Physics or Physics for Future Presidents. Because these fields have access to deep and challenging questions with demonstrably correct answers, the ability to answer these questions (a combination of cognitive ability and knowledge) is an obviously real and useful construct. See earlier post The Differences are Enormous:

Luis Alvarez laid it out bluntly:

The world of mathematics and theoretical physics is hierarchical. That was my first exposure to it. There's a limit beyond which one cannot progress. The differences between the limiting abilities of those on successively higher steps of the pyramid are enormous.

... People who work in "soft" fields (even in science) don't seem to understand this stark reality. I believe it is because their fields do not have ready access to right and wrong answers to deep questions. When those are available, huge differences in cognitive power are undeniable, as is the utility of this power.

Thought experiment for physicists: imagine a professor throwing copies of Jackson's Classical Electrodynamics at a group of students with the order, "Work out the last problem in each chapter and hand in your solutions to me on Monday!" I suspect that this exercise produces a highly useful rank ordering within the group, with huge differences in number of correct solutions.

Tetlock podcast: expert predictions

I recently came across this excellent talk (podcast number 84 on the list at the link) by Philip Tetlock about his research on expert prediction.

Putting aside the fox vs hedgehog dichotomy, I think the main takeaway is that "expert" predictions are no better than those of well-informed ordinary people, and barely outperform simple algorithms.

Longnow.org: ... Tetlock took advantage of getting tenure to start a long-term research project now 18 years old to examine in detail the outcomes of expert political forecasts about international affairs. He studied the aggregate accuracy of 284 experts making 28,000 forecasts, looking for pattern in their comparative success rates. Most of the findings were negative— conservatives did no better or worse than liberals; optimists did no better or worse than pessimists. Only one pattern emerged consistently.

“How you think matters more than what you think.”

It’s a matter of judgement style, first expressed by the ancient Greek warrior poet Archilochus: “The fox knows many things; the hedgehog one great thing.” The idea was later expanded by essayist Isaiah Berlin. In Tetlock’s interpretation, Hedgehogs have one grand theory (Marxist, Libertarian, whatever) which they are happy to extend into many domains, relishing its parsimony, and expressing their views with great confidence. Foxes, on the other hand are skeptical about grand theories, diffident in their forecasts, and ready to adjust their ideas based on actual events.

The aggregate success rate of Foxes is significantly greater, Tetlock found, especially in short-term forecasts. And Hedgehogs routinely fare worse than Foxes, especially in long-term forecasts. They even fare worse than normal attention-paying dilletantes — apparently blinded by their extensive expertise and beautiful theory. Furthermore, Foxes win not only in the accuracy of their predictions but also the accuracy of the likelihood they assign to their predictions— in this they are closer to the admirable discipline of weather forecasters.

The value of Hedgehogs is that they occasionally get right the farthest-out predictions— civil war in Yugoslavia, Saddam’s invasion of Kuwait, the collapse of the Internet Bubble. But that comes at the cost of a great many wrong far-out predictions— Dow 36,000, global depression, nuclear attack by developing nations.

Hedgehogs annoy only their political opposition, while Foxes annoy across the political spectrum, in part because the smartest Foxes cherry-pick idea fragments from the whole array of Hedgehogs.

Bottom line… The political expert who bores you with an cloud of “howevers” is probably right about what’s going to happen. The charismatic expert who exudes confidence and has a great story to tell is probably wrong.

And to improve the quality of your own predictions, keep brutally honest score. Enjoy being wrong, admitting to it and learning from it, as much as you enjoy being right.

See also Intellectual honesty: how much do we know?

Expert predictions

How good are "experts" at making accurate predictions? Much worse than you think, says psychology professor Philip Tetlock (Haas School of Business at UC Berkeley) in his new book Expert Political Judgment: How Good Is It? How Can We Know? (See New Yorker review.) In detailed studies, in which "experts" were asked to make forecasts about the future (predicting which of three possible futures would occur), it was found that the "experts" did no better than well-informed non-experts! As Tetlock says, “We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly,” he reports. “In this age of academic hyperspecialization, there is no reason for supposing that contributors to top journals—distinguished political scientists, area study specialists, economists, and so on—are any better than journalists or attentive readers of the New York Times in ‘reading’ emerging situations.”

Now, I expect the performance of scientific experts to be somewhat better. Questions like "How hot will that spacecraft get while in orbit around Mercury?" or "How many CPU cycles will it take to compute that integral?" are ones where predictions of real experts will far outperform those of lay people. I guess there is something fundamentally different about scientific versus non-scientific expertise? The last bit below about predicting freshman academic performance is amazing (but not unexpected). Let a simple one or two parameter model pick your freshman class :-)

Finally, what type of "expert" would you trust to run your money (make investment predictions)? As Jim Simons said: "The advantage scientists bring into the game is less their mathematical or computational skills than their ability to think scientifically. They are less likely to accept an apparent winning strategy that might be a mere statistical fluke." In other words, they know when they know something, while others might just be fooling themselves ;-)

New Yorker: Tetlock is a psychologist—he teaches at Berkeley—and his conclusions are based on a long-term study that he began twenty years ago. He picked two hundred and eighty-four people who made their living “commenting or offering advice on political and economic trends,” and he started asking them to assess the probability that various things would or would not come to pass, both in the areas of the world in which they specialized and in areas about which they were not expert. Would there be a nonviolent end to apartheid in South Africa? Would Gorbachev be ousted in a coup? Would the United States go to war in the Persian Gulf? Would Canada disintegrate? (Many experts believed that it would, on the ground that Quebec would succeed in seceding.) And so on. By the end of the study, in 2003, the experts had made 82,361 forecasts.

...Tetlock got a statistical handle on his task by putting most of the forecasting questions into a “three possible futures” form. The respondents were asked to rate the probability of three alternative outcomes: the persistence of the status quo, more of something (political freedom, economic growth), or less of something (repression, recession). And he measured his experts on two dimensions: how good they were at guessing probabilities (did all the things they said had an x per cent chance of happening happen x per cent of the time?), and how accurate they were at predicting specific outcomes. The results were unimpressive. On the first scale, the experts performed worse than they would have if they had simply assigned an equal probability to all three outcomes—if they had given each possible future a thirty-three-per-cent chance of occurring. Human beings who spend their lives studying the state of the world, in other words, are poorer forecasters than dart-throwing monkeys, who would have distributed their picks evenly over the three choices.

Tetlock also found that specialists are not significantly more reliable than non-specialists in guessing what is going to happen in the region they study.

...“Expert Political Judgment” is just one of more than a hundred studies that have pitted experts against statistical or actuarial formulas, and in almost all of those studies the people either do no better than the formulas or do worse. In one study, college counsellors were given information about a group of high-school students and asked to predict their freshman grades in college. The counsellors had access to test scores, grades, the results of personality and vocational tests, and personal statements from the students, whom they were also permitted to interview. Predictions that were produced by a formula using just test scores and grades were more accurate.

Taleb podcast: What do we know?

Here is an excellent talk by Nassim Taleb, hedge fund manager and author of the book Fooled by Randomness, which I highly recommend. Taleb addresses the prediction problem: how do you evaluate your knowledge of the world, other than by testing your ability to make predictions about what will happen next? (Post-diction is too easy - one can always construct post-hoc stories which are consistent with the data. Sorry, historians :-) He then notes that in certain fields like finance, economics and social science, the accuracy of predictions, when carefully studied, is dismal. (See my earlier discussion of Tetlock's research, which confirms this in a quantitative way. Tetlock had to work hard at this, since he looked at softer non-quantitative predictions as in foreign affairs. If you stick to quantitative predictions, like of equity or commodity prices, it is much easier to see that prognosticators are terrible.)

Feynman once said, holding up his fist and rotating it as if it were a charged sphere or something, "Physics is about answering the question: if I do this, what happens next?" I think this is very much in the spirit of Taleb's viewpoint.

One nice experiment Taleb describes shows how overconfident we are in our ability to predict the future.

Ask a group of people to make a prediction -- for example, how many Corollas will Toyota sell next (or last) year? We're not interested in the central values of their predictions. We're more interested in their understanding of its accuracy. So we say, give me a range that covers the 98 percent confidence interval. That is, give me a range of how many Corollas were sold last year, with the real value somewhere in that range at 98 percent confidence. Even if you know nothing about the auto industry, you could incorporate this into your guess by choosing a large range (e.g., between 10,000 and 10 million).

However, people are systematically overconfident in the quality of their predictions - by at least an order of magnitude, says Taleb. Typically, their 98 percent confidence level prediction is more like a 60 percent confidence level prediction. In other words, if you try this experiment with 100 students who correctly understand their own state of knowledge, you would expect only about 2 students to choose ranges which don't include the actual value. Instead, what you find is that 40 or so of the ranges will not contain the correct number! (Their error estimate of 2% is a gross underestimate.)

Taleb claims that the worst performing groups on this kind of exercise (regardless of the prediction requested) are stock analysts and economists, probably because the two groups are selected for a systematic bias toward overconfidence in dealing with noisy data. I wonder how physicists would do? I often stress that in communicating some information to a colleague (e.g., "A neutrino with those properties is ruled out by LEP data"), it is useful to also include a confidence level ("I have thought carefully about the loopholes and have looked at the LEP analysis and am 99% confident what I just said to you is true"). Thus, rather than transmitting a single statement, it is better to transmit the statement plus a confidence estimate. The utility of the pair is dramatically greater than just the statement itself.

My feeling is that when it comes to discussing the implications of a particular experiment, physicists are trained to accurately understand the confidence intervals. However, when it comes to a question like "How likely is it that supersymmetry solves the hierarchy problem?" I suspect we are as overconfident as any other group in the accuracy of our predictions.

Slate Star Codex on Superforecasting

Scott Alexander (Slate Star Codex) on Philip Tetlock's Superforecasting:

Book review
Highlighted passages.

I especially liked this passage that Scott highlights:

When hospitals created cardiac care units to treat patients recovering from heart attacks, Cochrane proposed a randomized trial to determine whether the new units delivered better results than the old treatment, which was to send the patient home for monitoring and bed rest. Physicians balked. It was obvious the cardiac care units were superior, they said, and denying patients the best care would be unethical. But Cochrane was not a man to back down…he got his trial: some patients, randomly selected, were sent to the cardiac care units while others were sent home for monitoring and bed rest. Partway through the trial, Cochrane met with a group of the cardiologists who had tried to stop his experiment. He told them that he had preliminary results. The difference in outcomes between the two treatments was not statistically signficant, he emphasized, but it appeared that patients might do slightly better in the cardiac care units. “They were vociferous in their abuse: ‘Archie,’ they said, ‘we always thought you were unethical. You must stop the trial at once.'” But then Cochrane revealed he had played a little trick. He had reversed the results: home care had done slightly better than the cardiac units. “There was dead silence and I felt rather sick because they were, after all, my medical colleagues.”

[ Cochrane Collaboration ]    [ Bounded Cognition ]

See also Medical Science?

Perils of Prediction

Highly recommended podcast: Tim Harford (FT) at the LSE. Among the topics covered are Keynes' and Irving Fisher's performance as investors, and Philip Tetlock's IARPA-sponsored Good Judgement Project, meant to evaluate expert prediction of complex events. Project researchers (psychologists) find that "actively open-minded thinkers" (those who are willing to learn from those that disagree with them) perform best. Unfortunately there are no real "super-predictors" -- just some who are better than others, and have better calibration (accurate confidence estimates).

Rule Britannia

Dom has seized the controls, but now has to operate the giant robot.

This is a unique situation: someone who understands the power of modern technology, scientific decision-making, high cognitive ability, and high functioning organizations, has significant influence in the government of one of the great nations of the world.

Please consider applying for one of these positions. Dom is a special leader -- open to new ideas and to maverick personalities, loyal to his team, and a genuinely humble and good person.

This could be a once in a lifetime opportunity to make a positive impact in the world.

High skill immigration is one of the priorities for the new UK government. You do not need to be a UK citizen or permanent resident to be considered for these positions.

...we’re hiring data scientists, project managers, policy experts, assorted weirdos...

There are many brilliant people in the civil service and politics. Over the past five months the No10 political team has been lucky to work with some fantastic officials. But there are also some profound problems at the core of how the British state makes decisions. This was seen by pundit-world as a very eccentric view in 2014. It is no longer seen as eccentric. ...

Now there is a confluence of: a) Brexit requires many large changes in policy and in the structure of decision-making, b) some people in government are prepared to take risks to change things a lot, and c) a new government with a significant majority and little need to worry about short-term unpopularity while trying to make rapid progress with long-term problems.

There is a huge amount of low hanging fruit — trillion dollar bills lying on the street — in the intersection of:

the selection, education and training of people for high performance,

the frontiers of the science of prediction data science,

AI and cognitive technologies (e.g Seeing Rooms, `authoring tools designed for arguing from evidence’, Tetlock/IARPA prediction tournaments that could easily be extended to consider ‘clusters’ of issues around themes like Brexit to improve policy and project management)

communication (e.g. Cialdini)

decision-making institutions at the apex of government.

We want to hire an unusual set of people with different skills and backgrounds to work in Downing Street with the best officials, some as spads and perhaps some as officials. If you are already an official and you read this blog and think you fit one of these categories, get in touch.

The categories are roughly:

Data scientists and software developers
Economists
Policy experts
Project managers
Communication experts
Junior researchers one of whom will also be my personal assistant
Weirdos and misfits with odd skills

[ Please click through and read the whole post on Dom's blog ]

See also Now it can be told: Dominic Cummings and the Conservative victory 2019.

Note Added: Some of the media takes on Dom's job ad are extremely uncharitable. They (and the people they quote) assume Dom is entirely naive about when mathematical and computational methods might be useful, and when they might not. I suggest these people study his other writing carefully. For example:

More important than technology is the mindset – the hard discipline of obeying Richard Feynman’s advice: ‘The most important thing is not to fool yourself and you are the easiest person to fool.’ They [quant types] were a hard floor on ‘fooling yourself’ and I empowered them to challenge everybody including me. They [quant types] saved me from many bad decisions even though they had zero experience in politics and they forced me to change how I made important decisions like what got what money. We either operated scientifically or knew we were not, which is itself very useful knowledge.

Does this sound like a person who does not understand both the strengths and limitations of data science, statistics, careful epistemology, etc. in modern politics? Underestimate him at your peril...

Freedom of Speech and Intellectual Diversity on Campus (MSU virtual conference)

The LeFrak Forum On Science, Reason, and Modern Democracy 

Department of Political Science 

Michigan State University 

Register here!

 

Thursday, April 8 -- Saturday, April 10; on ZOOM 

Conference Program: 

Keynote Address - Thursday, April 8, 

5:00-6:30pm EST 

Randall Kennedy, "The Race Question and Freedom of Expression." 

Randall Kennedy is the Michael R. Klein Professor at Harvard Law School, preeminent authority on the First Amendment in its relation to the American struggle for civil rights.

 

Day One: Intellectual Diversity - Friday, April 9  

11:30am - 1:00pm EST 

Panel 1: What are the empirical facts about lack of intellectual diversity in academia and what are the causes of existing imbalances? 

Paper: Lee Jussim, Distinguished Professor and Chair, Department of Psychology, Rutgers University, author of The Politics of Social Psychology. 

Discussant: Philip Tetlock, Annenberg University Professor, University of Pennsylvania, author of “Why so few conservatives and should we care?” and Cory Clark, Visiting Scholar, Department of Psychology, University of Pennsylvania, author of “Partisan Bias and its Discontents.” 

2:00pm - 3:30pm EST 

Panel 2: In what precise ways and to what degree is this imbalance a problem? 

Paper: Joshua Dunn, Professor and Chair, Department of Political Science, University of Colorado, co-author of Passing on the Right: Conservative Professors in the Progressive University. 

Discussant: Amna Khalid, Associate Professor of History, Carleton College, author of “Not A Vast Right-Wing Conspiracy: Why Left-Leaning Faculty Should Care About Threats to Free Expression on Campus." 

4:00pm - 5:45pm EST 

Panel 3: What is To Be Done? 

Paper: Musa Al-Gharbi, Paul F. Lazarsfeld Fellow in Sociology, Columbia University and Managing Editor, Heterodox Academy, author of “Why Care About Ideological Diversity in Social Research? The Definitive Response.” 

Paper: Conor Friedersdorf, Staff writer at The Atlantic and frequent contributor to its special series “The Speech Wars,” author of “Free Speech Will Survive This Moment.”

 

Day Two: Freedom of Speech - Saturday, April 10 

11:30am - 1:00pm EST 

Panel 1: An empirical accounting of the recent challenges to free speech on campus from left and right. What is the true character of the problem or problems here and do they constitute a “crisis”? 

Paper: Jonathan Marks, Professor and Chair, Department of Politics and International Relations, Ursinus College, author of Let's Be Reasonable: A Conservative Case for Liberal Education. 

Respondent: April Kelly-Woessner, Dean of the School of Public Service and Professor of Political Science at Elizabethtown College, author of The Still Divided Academy 

2:00pm - 3:45pm EST 

Panel 2: But is Free speech, as traditionally interpreted, even the right ideal? -- a Debate 

Ulrich Baer, University Professor of Comparative Literature, German, and English, NYU, author of What Snowflakes Get Right: Free Speech and Truth on Campus 

Keith Whittington, Professor of Politics, Princeton University, author of Speak Freely: Why Universities Must Defend Free Speech. 

4:30pm - 6:15pm EST  

Panel 3: What is To Be Done? 

Paper: Nancy Costello, Associate Clinical Professor of Law, MSU. Founder and Director of the First Amendment Law Clinic -- the only law clinic in the nation devoted to the defense of student press rights. Also, Director of the Free Expression Online Library and Resource Center. 

Paper: Jonathan Friedman, Project Director for campus free speech at PEN America – “a program of advocacy, analysis, and outreach in the national debate around free speech and inclusion at colleges and universities.”

The illusion of skill

Daniel Kahneman claims that differences in the performance of professional investors are mostly due to luck, whereas compensation is awarded as if differences are due to skill. Most alpha is fake alpha.

This of course raises all sorts of questions about why such people are allowed to become so extravagantly wealthy. The usual argument is that their investment decisions lead to more efficient resource allocation in the economy. (They are a "necessary evil" of a capitalist market system that benefits all of us :-) But if the decisions of the highest paid professionals are no better than those of average professionals, we could replace the services of the highest earners at much lower cost (or cap their salaries or impose high marginal tax rates) without negatively impacting the overall quality of decisions or the efficiency of the economy.








The article is worth reading in its entirety.

NYTimes: ... No one in the firm seemed to be aware of the nature of the game that its stock pickers were playing. The advisers themselves felt they were competent professionals performing a task that was difficult but not impossible, and their superiors agreed. On the evening before the seminar, Richard Thaler and I had dinner with some of the top executives of the firm, the people who decide on the size of bonuses. We asked them to guess the year-to-year correlation in the rankings of individual advisers. They thought they knew what was coming and smiled as they said, “not very high” or “performance certainly fluctuates.” It quickly became clear, however, that no one expected the average correlation to be zero.

What we told the directors of the firm was that, at least when it came to building portfolios, the firm was rewarding luck as if it were skill. This should have been shocking news to them, but it was not. There was no sign that they disbelieved us. How could they? After all, we had analyzed their own results, and they were certainly sophisticated enough to appreciate their implications, which we politely refrained from spelling out. We all went on calmly with our dinner, and I am quite sure that both our findings and their implications were quickly swept under the rug and that life in the firm went on just as before. The illusion of skill is not only an individual aberration; it is deeply ingrained in the culture of the industry. Facts that challenge such basic assumptions — and thereby threaten people’s livelihood and self-esteem — are simply not absorbed. The mind does not digest them. This is particularly true of statistical studies of performance, which provide general facts that people will ignore if they conflict with their personal experience.

The next morning, we reported the findings to the advisers, and their response was equally bland. Their personal experience of exercising careful professional judgment on complex problems was far more compelling to them than an obscure statistical result. When we were done, one executive I dined with the previous evening drove me to the airport. He told me, with a trace of defensiveness, “I have done very well for the firm, and no one can take that away from me.” I smiled and said nothing. But I thought, privately: Well, I took it away from you this morning. If your success was due mostly to chance, how much credit are you entitled to take for it?

From the comments.

If you read the whole article, you see that Kahneman does believe in skill. For example, his studies show that some doctors are better at diagnosis than others. I am also sure that some entrepreneurs or some physicists or some athletes are better than others. (Although in the case of entrepreneurs it would be very hard to demonstrate statistically since outcomes are noisy and the number of attempts per entrepreneur is relatively small.)

But there may be areas where *the differences between high level professionals* (e.g., people who have been hired to run money, have top MBAs or graduate degrees, etc.) are statistically seen to be mostly due to luck. This has already been convincingly demonstrated for pundits or analysts of complex world events by Tetlock's studies of expertise. (You can find several posts on this blog on the topic.) Whether it's true of money managers (or even big company CEOs) is controversial. If you argue the skill side, I'd like to see *your* statistical evidence, not just repetition of your priors (again and again).

Expert predictions

In all areas of human activity, even the skill dominated ones, luck plays a big factor. This is a good argument for redistribution -- almost every successful person owes some of their success to luck.

and

It seems to me that the 20th century trend in democracies is toward greater redistribution: social safety nets, guaranteed minimum income, etc. People have been conditioned to believe these are aspects of a just society.

The question is: what is the optimum level of redistribution? (Given a particular utility function for society.)

One argument is that we have to let the rich get rich in order to have strong economic growth. Too much redistribution means a smaller pie to split. But the Illusion of Skill argument (if correct) suggests that for some activities like finance a high marginal tax rate (say, which kicks in above the income of the *average* finance professional; this would then only affect the top earning financiers who, according to the argument are not adding any real value that the average guys can't also provide) would not negatively affect economic efficiency.

If people irrationally and incorrectly believe that only Harvard MDs are capable of treating pneumonia, and bid up their compensation to exorbitant levels (levels so high that the Harvard MDs begin exerting financial and political control over society as a whole), wouldn't it be better for society to impose a high tax rate on Harvard MDs, which kicks in above the income of other doctors with similar credentials (but who are not beneficiaries of the irrational belief)?

Worth a look or listen

Some recommendations from a bunch of content I consumed during recent travel.

The Black Swan by Nassim Taleb. I finally got around to reading this and recommend it highly. Physicists and others who are already familiar with nonlinear dynamics (chaos theory), the difference between Gaussian and power-law distributions, etc. will find the presentation slow and repetitious at times, but Taleb does have a lot of interesting insights. Particularly amusing: Chapter 10, The scandal of prediction, in which he recapitulates Philip Tetlock's results, chapter 17, which rails against the "Nobel" prize in economics, especially the one awarded to Merton and Scholes. I can't say I completely agree with Taleb on the (non)utility of modern finance theory. It's true that Gaussians underestimate the likelihood of rare events, but that is well known now and there are various ways to incorporate that into models (e.g., fat tails, stochastic vol). He's dismissive of these improvements in the book; it appears to me he's attacking a caricature from 10 years ago.

I also recommend a number of podcast interviews from the site Econtalk.org. Especially useful if you're going to be stuck on a plane, train or automobile. Some that I found especially good:

Taleb on the Black Swan (strange that the interviewer, an economist, didn't explore Taleb's extremely negative view of the profession! I guess they're both Hayekians so had some common ground :-)

Paul Romer on economic growth.

Ed Leamer on outsourcing and trade.

Vernon Smith on experimental economics.

Gregg Easterbrook on happiness and the American standard of living (we're 10x richer on average than 100 years ago!).

Bob Lucas on growth, poverty, monetary policy.