Success, Ability, and all that

I came across this nice discussion at LessWrong which is similar to my old post Success vs Ability. The illustration below shows why even a strong predictor of outcome is seldom able to pick out the very top performer: e.g., taller people are on average better at basketball, but the best player in the world is not the tallest; smarter people are on average better at making money, but the richest person in the world is not the smartest, etc.

This seems like a trivial point (as are most things, when explained clearly), however, it still eludes the vast majority. For example, in the Atlantic article I linked to in the earlier post Creative Minds, the neuroscientist professor who studies creative genius misunderstands the implications of the Terman study. She repeats the common claim that Terman's study fails to support the importance of high cognitive ability to "genius"-level achievement: none of the Termites won a Nobel prize, whereas Shockley and Alvarez, who narrowly missed the (verbally loaded) Stanford-Binet cut for the study, each won for work in experimental physics. But luck, drive, creativity, and other factors, all at least somewhat independent of intelligence, influence success in science. Combine this with the fact that there are exponentially more people a bit below the Terman cut than above it, and Terman's results do little more than confirm that cognitive ability is positively but not perfectly correlated with creative output.

In the SMPY study probability of having published a literary work or earned a patent was increasing with ability even within the top 1%. The "IQ over 120 doesn't matter" meme falls apart if one measures individual likelihood of success, as opposed to the total number of individuals at, e.g., IQ 120 vs IQ 145, who have achieved some milestone.

It is plausible that, e.g., among top execs or scientists or engineers there are roughly equal numbers of IQ 120 and IQ 145 individuals (the actual numbers could vary depending on how the groups are defined). But the base population of the former group is 100 times that of the latter! (IQ 120 is about top 10% and IQ 145 is roughly top 0.1% in the population.) This means, e.g., that the probability that an IQ 145 person becomes a top scientist could be ~100x higher than for an IQ 120 person.

This topic came up last night in Hong Kong, at dinner with two hedge funders (Caltech/MIT guys with PhDs) who have had long careers in finance. Both observed that 20 years ago it was nearly impossible to predict which of their colleagues and peers would go on to make vast fortunes, as opposed to becoming merely rich.

Better to be lucky than good

Shorter Taleb (much of this was discussed in his first book, Fooled by Randomness):

Fat tails + nonlinear feedback means that the majority of successful traders were successful due to luck, not skill. It's painful to live in the shadow of such competitors.

What other fields are dominated by noisy feedback loops? See Success vs Ability , Nonlinearity and noisy outcomes , The illusion of skill and Fake alpha.

Why It is No Longer a Good Idea to Be in The Investment Industry 

Nassim N. Taleb 

Abstract: A spurious tail is the performance of a certain number of operators that is entirely caused by luck, what is called the “lucky fool” in Taleb (2001). Because of winner-take-all-effects (from globalization), spurious performance increases with time and explodes under fat tails in alarming proportions. An operator starting today, no matter his skill level, and ability to predict prices, will be outcompeted by the spurious tail. This paper shows the effect of powerlaw distributions on such spurious tail. The paradox is that increase in sample size magnifies the role of luck. 

... The “spurious tail” is therefore the number of persons who rise to the top for no reasons other than mere luck, with subsequent rationalizations, analyses, explanations, and attributions. The performance in the “spurious tail” is only a matter of number of participants, the base population of those who tried. Assuming a symmetric market, if one has for base population 1 million persons with zero skills and ability to predict starting Year 1, there should be 500K spurious winners Year 2, 250K Year 3, 125K Year 4, etc. One can easily see that the size of the winning population in, say, Year 10 depends on the size of the base population Year 1; doubling the initial population would double the straight winners. Injecting skills in the form of better-than-random abilities to predict does not change the story by much. 

Because of scalability, the top, say 300, managers get the bulk of the allocations, with the lion’s share going to the top 30. So it is obvious that the winner-take-all effect causes distortions ...

Conclusions: The “fooled by randomness” effect grows under connectivity where everything on the planet flows to the “top x”, where x is becoming a smaller and smaller share of the top participants. Today, it is vastly more acute than in 2001, at the time of publication of (Taleb 2001). But what makes the problem more severe than anticipated, and causes it to grow even faster, is the effect of fat tails. For a population composed of 1 million track records, fat tails multiply the threshold of spurious returns by between 15 and 30 times. 

Generalization: This condition affects any business in which prevail (1) some degree of fat-tailed randomness, and (2) winner-take-all effects in allocation. 

To conclude, if you are starting a career, move away from investment management and performance related lotteries as you will be competing with a swelling future spurious tail. Pick a less commoditized business or a niche where there is a small number of direct competitors. Or, if you stay in trading, become a market-maker.

Bonus question: what are the ramifications for tax and economic policies (i.e., meant to ensure efficiency and just outcomes) of the observation that a particular industry is noise dominated?

Douglas Hofstader at UO

A former physicist I went to grad school with, who now lives in Eugene, writes about a talk given by Doug Hofstader yesterday. Unfortunately I missed the talk, although I read Hofstader's autobiographical I am a Strange Loop a few years ago. Godel, Escher, Bach made a big impression on me in high school. I actually considered going into AI, until I discovered there was no there there ;-)

The speaker did not give a talk so much on physics but rather on his personal experiences up through physics grad school at the University of Oregon, many years ago. His father was a famous physicist, but he was more interested in math as a young boy, especially number theory. He majored in math at Stanford, and then started working on a degree in theoretical particle physics at Berkeley. But he found the work being done there "ugly" and way above his abilities. Moreover, this was during the Vietnam War, and the politics on campus back then was too intense for him, although he was sympathetic. He ended up transferring to the UO physics department, where he at least found the atmosphere much more laid back. But he still found theoretical physics to be too hard for him, and too "ugly" (too many unjustified assumptions motivated merely by flimsy mathematical justifications). At one point he was so disgusted with his field that he quit. He ended up doing theoretical solid state physics, even though he had originally felt like this was little more than engineering. To cut to the chase, he started his talk recounting some experimental mathematics he did while still a teenager, using what was then Stanford's only computer. And he connected in telling part of his life story, he ended up coming full circle to explain how his earlier interests as a boy connected to a significant paper he published towards the end of his graduate career. This paper concerned how the energy bands of crystals in a magnetic field depend on the strength of that field. A graph summarizing his results has been reproduced many times on the covers of solid state physics texts and physics journals, making him a kind of physics celebrity. What he discovered is known as the Hofstadter Butterfly and his name is Douglas Hofstadter.

This was a fascinating talk, not only because his experiences mirrored mine (and probably many of yours) to some extent, but also because he wrote the Pulitzer-Prize winning book Gödel, Escher, Bach, which came out and I ready while studying math in college. But he did not talk about his book, at least not directly, but rather about many of his failures proceeding his subsequent success with the book, become a professor of cognitive science, publishing other books, and succeeding Martin Gardner as the mathematics columnist for Scientific American. But afterwards I asked him about GEB, as the introduction to the book explains that he wrote a first version of the book while still a grad student at UO. It was a passion of love for him, but one that he saw was interfering with getting a doctorate, which he felt he needed to do in order to have a membership card into academia. So for a time he found ways to forbid himself from working on the book. (He would reduce his small food allowance for every hour he worked on the book, which could easily have led him to starvation.) Subsequently, he totally rewrote the book to be in the form that was published, but that is all sequel to the "failed" period of life he was describing.

If there was a moral to his talk -- it certainly wasn't about any deep physical or mathematical principles -- it was about how luck plays an important role in life, perhaps as much as innate ability, skill and hard work.

Hollywood genius

Physicist turned author and screenwriter Leonard Mlodinow has a nice article in the LA Times on the hit or miss nature of the movie industry. He recapitulates the myth of expertise as it applies to studio executives, whom he compares to dart throwing monkeys (a la fund managers in finance).

Mlodinow wrote a charming memoir about his time as a postdoc at Caltech in the early 1980s. Fresh from Berkeley, having written a PhD dissertation on the large-d expansion (d is the number of dimensions), he was in over his head at Caltech, but found a friend and mentor in the ailing Richard Feynman.

We all understand that genius doesn't guarantee success, but it's seductive to assume that success must come from genius. As a former Hollywood scriptwriter, I understand the comfort in hiring by track record. Yet as a scientist who has taught the mathematics of randomness at Caltech, I also am aware that track records can deceive.

That no one can know whether a film will hit or miss has been an uncomfortable suspicion in Hollywood at least since novelist and screenwriter William Goldman enunciated it in his classic 1983 book "Adventures in the Screen Trade." If Goldman is right and a future film's performance is unpredictable, then there is no way studio executives or producers, despite all their swagger, can have a better track record at choosing projects than an ape throwing darts at a dartboard.

That's a bold statement, but these days it is hardly conjecture: With each passing year the unpredictability of film revenue is supported by more and more academic research.

That's not to say that a jittery homemade horror video could just as easily become a hit as, say, "Exorcist: The Beginning," which cost an estimated $80 million, according to Box Office Mojo, the source for all estimated budget and revenue figures in this story. Well, actually, that is what happened with "The Blair Witch Project" (1999), which cost the filmmakers a mere $60,000 but brought in $140 million—more than three times the business of "Exorcist." (Revenue numbers reflect only domestic receipts.)

What the research shows is that even the most professionally made films are subject to many unpredictable factors that arise during production and marketing, not to mention the inscrutable taste of the audience. It is these unknowns that obliterate the ability to foretell the box-office future.

But if picking films is like randomly tossing darts, why do some people hit the bull's-eye more often than others? For the same reason that in a group of apes tossing darts, some apes will do better than others. The answer has nothing to do with skill. Even random events occur in clusters and streaks.

...If the mathematics is counterintuitive, reality is even worse, because a funny thing happens when a random process such as the coin-flipping experiment is actually carried out: The symmetry of fairness is broken and one of the films becomes the winner. Even in situations like this, in which we know there is no "reason" that the coin flips should favor one film over the other, psychologists have shown that the temptation to concoct imagined reasons to account for skewed data and other patterns is often overwhelming.

...Actors in Hollywood understand best that the industry runs on luck. As Bruce Willis once said, "If you can find out why this film or any other film does any good, I'll give you all the money I have." (For the record, the film to which he referred, 1993's "Striking Distance," didn't do any good.) Willis understands the unpredictability of the film business not simply because he's had box-office highs and lows. He knows that random events fueled his career from the beginning, and his story offers another case in point...

Success vs ability

The figure above illustrates the correlation between two variables, let us say success and ability. Each point represents an individual whose level of success and ability are shown on the vertical and horizontal axes, respectively. In the figure, the correlation is high, but not 100%.

For example, in American football the ability axis might represent the quantities obsessively tracked by NFL scouts: sprinting speed (40 yard dash time), natural strength (bench press), etc., while the vertical axis represents actual output, like passes caught or rushing yards gained. In real life, output is never purely determined by a single, or even several, input ability or abilities. If nothing else, luck ensures that the correlation is imperfect. Sports fans know that the fastest wide receiver isn't necessarily the best, nor the tallest basketball center the most productive, even if being fast or tall confer specific advantages. In the figure, the most able individual is not the most successful. They are seldom the same individual unless the correlation is 100%

In science or academia, we might take the horizontal axis to represent raw intellectual ability. The graph tells us to expect that the smartest person is not necessarily the most successful. It also suggests a population of successful but insecure people (the upper right dots above the fit line -- they are dumber than peers of similar accomplishment) and a population of smart people who are bitter about their unrecognized genius (dots on far right below the fit line -- they are smarter than peers of similar accomplishment).