Warren Hatch on Seeing the Future in the Era of COVID-19: Manifold Episode #50

Steve and Corey talk to Warren Hatch, President and CEO of Good Judgment Inc. Warren explains what makes someone a good forecaster and how the ability to integrate and assess information allows cognitively diverse teams to outperform prediction markets. The hosts express skepticism about whether the incentives at work in large organizations would encourage the adoption of approaches that might lead to better forecasts. Warren describes the increasing depth of human-computer collaboration in forecasting. Steve poses the long-standing problem of assessing alpha in finance and Warren suggests that the emerging alpha-brier metric, linking process and outcome, might shed light on the issue. The episode ends with Warren describing Good Judgment’s open invitation to self-identified experts to join a new COVID forecasting platform.

Transcript

Good Judgment Inc.

Good Judgment Open

Superforecasting: The Art and Science of Prediction

Noriel Roubini (Wikipedia)


man·i·fold /ˈmanəˌfōld/ many and various.

In mathematics, a manifold is a topological space that locally resembles Euclidean space near each point.

Steve Hsu and Corey Washington have been friends for almost 30 years, and between them hold PhDs in Neuroscience, Philosophy, and Theoretical Physics. Join them for wide ranging and unfiltered conversations with leading writers, scientists, technologists, academics, entrepreneurs, investors, and more.

Steve Hsu is VP for Research and Professor of Theoretical Physics at Michigan State University. He is also a researcher in computational genomics and founder of several Silicon Valley startups, ranging from information security to biotech. Educated at Caltech and Berkeley, he was a Harvard Junior Fellow and held faculty positions at Yale and the University of Oregon before joining MSU.

Corey Washington is Director of Analytics in the Office of Research and Innovation at Michigan State University. He was educated at Amherst College and MIT before receiving a PhD in Philosophy from Stanford and a PhD in a Neuroscience from Columbia. He held faculty positions at the University Washington and the University of Maryland. Prior to MSU, Corey worked as a biotech consultant and is founder of a medical diagnostics startup.

Better to be lucky than good

Shorter Taleb (much of this was discussed in his first book, Fooled by Randomness):

Fat tails + nonlinear feedback means that the majority of successful traders were successful due to luck, not skill. It's painful to live in the shadow of such competitors.

What other fields are dominated by noisy feedback loops? See Success vs Ability , Nonlinearity and noisy outcomes , The illusion of skill and Fake alpha.

Why It is No Longer a Good Idea to Be in The Investment Industry 

Nassim N. Taleb 

Abstract: A spurious tail is the performance of a certain number of operators that is entirely caused by luck, what is called the “lucky fool” in Taleb (2001). Because of winner-take-all-effects (from globalization), spurious performance increases with time and explodes under fat tails in alarming proportions. An operator starting today, no matter his skill level, and ability to predict prices, will be outcompeted by the spurious tail. This paper shows the effect of powerlaw distributions on such spurious tail. The paradox is that increase in sample size magnifies the role of luck. 

... The “spurious tail” is therefore the number of persons who rise to the top for no reasons other than mere luck, with subsequent rationalizations, analyses, explanations, and attributions. The performance in the “spurious tail” is only a matter of number of participants, the base population of those who tried. Assuming a symmetric market, if one has for base population 1 million persons with zero skills and ability to predict starting Year 1, there should be 500K spurious winners Year 2, 250K Year 3, 125K Year 4, etc. One can easily see that the size of the winning population in, say, Year 10 depends on the size of the base population Year 1; doubling the initial population would double the straight winners. Injecting skills in the form of better-than-random abilities to predict does not change the story by much. 

Because of scalability, the top, say 300, managers get the bulk of the allocations, with the lion’s share going to the top 30. So it is obvious that the winner-take-all effect causes distortions ...

Conclusions: The “fooled by randomness” effect grows under connectivity where everything on the planet flows to the “top x”, where x is becoming a smaller and smaller share of the top participants. Today, it is vastly more acute than in 2001, at the time of publication of (Taleb 2001). But what makes the problem more severe than anticipated, and causes it to grow even faster, is the effect of fat tails. For a population composed of 1 million track records, fat tails multiply the threshold of spurious returns by between 15 and 30 times. 

Generalization: This condition affects any business in which prevail (1) some degree of fat-tailed randomness, and (2) winner-take-all effects in allocation. 

To conclude, if you are starting a career, move away from investment management and performance related lotteries as you will be competing with a swelling future spurious tail. Pick a less commoditized business or a niche where there is a small number of direct competitors. Or, if you stay in trading, become a market-maker.

Bonus question: what are the ramifications for tax and economic policies (i.e., meant to ensure efficiency and just outcomes) of the observation that a particular industry is noise dominated?

The illusion of skill

Daniel Kahneman claims that differences in the performance of professional investors are mostly due to luck, whereas compensation is awarded as if differences are due to skill. Most alpha is fake alpha.

This of course raises all sorts of questions about why such people are allowed to become so extravagantly wealthy. The usual argument is that their investment decisions lead to more efficient resource allocation in the economy. (They are a "necessary evil" of a capitalist market system that benefits all of us :-) But if the decisions of the highest paid professionals are no better than those of average professionals, we could replace the services of the highest earners at much lower cost (or cap their salaries or impose high marginal tax rates) without negatively impacting the overall quality of decisions or the efficiency of the economy.








The article is worth reading in its entirety.

NYTimes: ... No one in the firm seemed to be aware of the nature of the game that its stock pickers were playing. The advisers themselves felt they were competent professionals performing a task that was difficult but not impossible, and their superiors agreed. On the evening before the seminar, Richard Thaler and I had dinner with some of the top executives of the firm, the people who decide on the size of bonuses. We asked them to guess the year-to-year correlation in the rankings of individual advisers. They thought they knew what was coming and smiled as they said, “not very high” or “performance certainly fluctuates.” It quickly became clear, however, that no one expected the average correlation to be zero.

What we told the directors of the firm was that, at least when it came to building portfolios, the firm was rewarding luck as if it were skill. This should have been shocking news to them, but it was not. There was no sign that they disbelieved us. How could they? After all, we had analyzed their own results, and they were certainly sophisticated enough to appreciate their implications, which we politely refrained from spelling out. We all went on calmly with our dinner, and I am quite sure that both our findings and their implications were quickly swept under the rug and that life in the firm went on just as before. The illusion of skill is not only an individual aberration; it is deeply ingrained in the culture of the industry. Facts that challenge such basic assumptions — and thereby threaten people’s livelihood and self-esteem — are simply not absorbed. The mind does not digest them. This is particularly true of statistical studies of performance, which provide general facts that people will ignore if they conflict with their personal experience.

The next morning, we reported the findings to the advisers, and their response was equally bland. Their personal experience of exercising careful professional judgment on complex problems was far more compelling to them than an obscure statistical result. When we were done, one executive I dined with the previous evening drove me to the airport. He told me, with a trace of defensiveness, “I have done very well for the firm, and no one can take that away from me.” I smiled and said nothing. But I thought, privately: Well, I took it away from you this morning. If your success was due mostly to chance, how much credit are you entitled to take for it?

From the comments.

If you read the whole article, you see that Kahneman does believe in skill. For example, his studies show that some doctors are better at diagnosis than others. I am also sure that some entrepreneurs or some physicists or some athletes are better than others. (Although in the case of entrepreneurs it would be very hard to demonstrate statistically since outcomes are noisy and the number of attempts per entrepreneur is relatively small.)

But there may be areas where *the differences between high level professionals* (e.g., people who have been hired to run money, have top MBAs or graduate degrees, etc.) are statistically seen to be mostly due to luck. This has already been convincingly demonstrated for pundits or analysts of complex world events by Tetlock's studies of expertise. (You can find several posts on this blog on the topic.) Whether it's true of money managers (or even big company CEOs) is controversial. If you argue the skill side, I'd like to see *your* statistical evidence, not just repetition of your priors (again and again).

Expert predictions

In all areas of human activity, even the skill dominated ones, luck plays a big factor. This is a good argument for redistribution -- almost every successful person owes some of their success to luck.

and

It seems to me that the 20th century trend in democracies is toward greater redistribution: social safety nets, guaranteed minimum income, etc. People have been conditioned to believe these are aspects of a just society.

The question is: what is the optimum level of redistribution? (Given a particular utility function for society.)

One argument is that we have to let the rich get rich in order to have strong economic growth. Too much redistribution means a smaller pie to split. But the Illusion of Skill argument (if correct) suggests that for some activities like finance a high marginal tax rate (say, which kicks in above the income of the *average* finance professional; this would then only affect the top earning financiers who, according to the argument are not adding any real value that the average guys can't also provide) would not negatively affect economic efficiency.

If people irrationally and incorrectly believe that only Harvard MDs are capable of treating pneumonia, and bid up their compensation to exorbitant levels (levels so high that the Harvard MDs begin exerting financial and political control over society as a whole), wouldn't it be better for society to impose a high tax rate on Harvard MDs, which kicks in above the income of other doctors with similar credentials (but who are not beneficiaries of the irrational belief)?

Fear and loathing of the plutocracy

Did Paulson and Treasury deliberately overpay for shares in Goldman and eight other banks? What did taxpayers get for their $125 billion, versus what Buffet got for his $5 billion Goldman investment just a few weeks earlier? See Black-Scholes analysis here.

Will banks like Citi pay out bonuses using bailout funds? Citi plans to distribute $26 billion after receiving $25 billion from you and me. Will the public let them get away with it? Given the tremendous value destruction they've caused, on Wall Street and beyond, how can top executives at these companies justify any bonus compensation for themselves?

After leaving the Clinton administration, how did Rahm Emanuel make $16 million in two years (at Wasserstein Perella) with no prior business experience? Did you really think Obama was going to be a radical left president?

Michael Lewis, Bloomberg commentary:

It may still take awhile before Wall Street finally accepts that it won't get paid.

At the moment, as their bony fingers fondle the new taxpayer loot, the firms appear to believe that they might still fool the public into thinking that bonus money isn't taxpayer money.

``We've responded appropriately to the attorney general's request for information about 2008 bonus pools,'' a Citigroup Inc. spokeswoman told Bloomberg News recently, ``and confirmed that we will not use TARP funds for compensation.'' But as the Bloomberg report noted, ``she declined to elaborate.''

As well she might! For if the Citigroup spokeswoman had elaborated she would have needed to say something like this: ``We're still trying to figure out how the $25 billion we've already taken of taxpayers' money has nothing to do with the $26 billion we're planning to hand out to our highly paid employees in 2008 (up 4 percent from 2007!). But it's a tricky problem because, when you think about it, it's all the same money.''

...If you are one of those people currently sitting inside a big Wall Street firm praying for some kind of bonus it may already have dawned on you that you need to rethink your approach. It's no longer any use to hint darkly that they had better fork over serious sticks or you'll bolt for Morgan Stanley. There's no point even in thinking up clever ways to make profits for your firm: who cares how much money you bring into Goldman Sachs if the U.S. Congress doesn't allow Goldman Sachs to pay bonuses?

The moment your firm accepted taxpayer money, you lost control of your money machine. ...

More from Michael Lewis at Portfolio Magazine (via Paul Kedrosky).


Figure from earlier post: Is the finance boom over?

Hollywood genius

Physicist turned author and screenwriter Leonard Mlodinow has a nice article in the LA Times on the hit or miss nature of the movie industry. He recapitulates the myth of expertise as it applies to studio executives, whom he compares to dart throwing monkeys (a la fund managers in finance).

Mlodinow wrote a charming memoir about his time as a postdoc at Caltech in the early 1980s. Fresh from Berkeley, having written a PhD dissertation on the large-d expansion (d is the number of dimensions), he was in over his head at Caltech, but found a friend and mentor in the ailing Richard Feynman.

We all understand that genius doesn't guarantee success, but it's seductive to assume that success must come from genius. As a former Hollywood scriptwriter, I understand the comfort in hiring by track record. Yet as a scientist who has taught the mathematics of randomness at Caltech, I also am aware that track records can deceive.

That no one can know whether a film will hit or miss has been an uncomfortable suspicion in Hollywood at least since novelist and screenwriter William Goldman enunciated it in his classic 1983 book "Adventures in the Screen Trade." If Goldman is right and a future film's performance is unpredictable, then there is no way studio executives or producers, despite all their swagger, can have a better track record at choosing projects than an ape throwing darts at a dartboard.

That's a bold statement, but these days it is hardly conjecture: With each passing year the unpredictability of film revenue is supported by more and more academic research.

That's not to say that a jittery homemade horror video could just as easily become a hit as, say, "Exorcist: The Beginning," which cost an estimated $80 million, according to Box Office Mojo, the source for all estimated budget and revenue figures in this story. Well, actually, that is what happened with "The Blair Witch Project" (1999), which cost the filmmakers a mere $60,000 but brought in $140 million—more than three times the business of "Exorcist." (Revenue numbers reflect only domestic receipts.)

What the research shows is that even the most professionally made films are subject to many unpredictable factors that arise during production and marketing, not to mention the inscrutable taste of the audience. It is these unknowns that obliterate the ability to foretell the box-office future.

But if picking films is like randomly tossing darts, why do some people hit the bull's-eye more often than others? For the same reason that in a group of apes tossing darts, some apes will do better than others. The answer has nothing to do with skill. Even random events occur in clusters and streaks.

...If the mathematics is counterintuitive, reality is even worse, because a funny thing happens when a random process such as the coin-flipping experiment is actually carried out: The symmetry of fairness is broken and one of the films becomes the winner. Even in situations like this, in which we know there is no "reason" that the coin flips should favor one film over the other, psychologists have shown that the temptation to concoct imagined reasons to account for skewed data and other patterns is often overwhelming.

...Actors in Hollywood understand best that the industry runs on luck. As Bruce Willis once said, "If you can find out why this film or any other film does any good, I'll give you all the money I have." (For the record, the film to which he referred, 1993's "Striking Distance," didn't do any good.) Willis understands the unpredictability of the film business not simply because he's had box-office highs and lows. He knows that random events fueled his career from the beginning, and his story offers another case in point...