Saturday, July 26, 2014

Success, Ability, and all that

I came across this nice discussion at LessWrong which is similar to my old post Success vs Ability. The illustration below shows why even a strong predictor of outcome is seldom able to pick out the very top performer: e.g., taller people are on average better at basketball, but the best player in the world is not the tallest; smarter people are on average better at making money, but the richest person in the world is not the smartest, etc.


This seems like a trivial point (as are most things, when explained clearly), however, it still eludes the vast majority. For example, in the Atlantic article I linked to in the earlier post Creative Minds, the neuroscientist professor who studies creative genius misunderstands the implications of the Terman study. She repeats the common claim that Terman's study fails to support the importance of high cognitive ability to "genius"-level achievement: none of the Termites won a Nobel prize, whereas Shockley and Alvarez, who narrowly missed the (verbally loaded) Stanford-Binet cut for the study, each won for work in experimental physics. But luck, drive, creativity, and other factors, all at least somewhat independent of intelligence, influence success in science. Combine this with the fact that there are exponentially more people a bit below the Terman cut than above it, and Terman's results do little more than confirm that cognitive ability is positively but not perfectly correlated with creative output.


In the SMPY study probability of having published a literary work or earned a patent was increasing with ability even within the top 1%. The "IQ over 120 doesn't matter" meme falls apart if one measures individual likelihood of success, as opposed to the total number of individuals at, e.g., IQ 120 vs IQ 145, who have achieved some milestone. The base population of the former is 100 times that of the latter!

This topic came up last night in Hong Kong, at dinner with two hedge funders (Caltech/MIT guys with PhDs) who have had long careers in finance. Both observed that 20 years ago it was nearly impossible to predict which of their colleagues and peers would go on to make vast fortunes, as opposed to becoming merely rich.

43 comments:

Jon Claerbout said...

Few Muslims are international terrorists, but most international terrorists are Muslims. Are we talking probability functions, or sets?

Boy said...

"e.g., IQ 120 vs IQ 145 who have achieved some milestone. There are 100 times more of the former than then the latter!" mixup

Pumpkinperson.com said...

Excellent post. Over the years I've read every major IQ blogger on the internet, and none of them have made as much sense as you! You're one of those rare people who doesn't suffer from bounded cognition, .

Not only should we not expect the highest IQ person to be the most "successful". but we shouldn't even expect the highest IQ person to be the most intelligent, given the imperfect correlation between IQ and g or intelligence.

Given the 0.65 correlation between IQ and academic success (whether defined as grades range corrected or years of education), we should expect one in a million academic achievers (Nobel prize winners) to have about a one in 2000 level IQ,

http://brainsize.wordpress.com/2014/07/12/the-iqs-of-academic-elites/

Conversely, one in a million IQ should perform at the one in 2000 level academically (i.e. maybe get into Harvard).

We should not expect anything spectacular from Terman type intellects (one in 100?) on average (a college degree, but not much beyond that).

Richard Seiter said...

It would be interesting to calculate likelihoods for the X (e.g. height, IQ) of the best Y (e.g. basketball player, wealthiest) for a given correlation. I wonder what the MLE would be and how it would compare to Bill Gates' IQ (since we can actually make a decent estimate given his SAT score). It would probably be even better to look at the top n instead.


The shape of that likelihood curve would be quite interesting as well. I would also like to see likelihoods for in top a% of Y given in top b% of X or at c% of X.


One nit. I think your last sentence is misleading. That may be true for people AT the Terman cutoff, but I think the phrase "Terman type intellects" more accurately means ABOVE the Terman cutoff. I think we should expect exceptional (don't know about spectacular, depends on population size) things from that group. As an aside I wish someone could replicate the Terman experiment with an IQ test that was less V dominated (did Shockley or Alvarez take any other tests that would estimate IQ? Anne Roe's papers should have later life data for Alvarez).

Pincher Martin said...

"We should not expect anything spectacular from Terman type intellects (one in 100?) on average (a college degree, but not much beyond that)."


At a time when more than thirty percent of Americans over the age of twenty-five have bachelor's degrees and more than ten percent have graduate degrees, we should expect much more than that.

Pincher Martin said...

"It would be interesting to calculate likelihoods for the X (e.g. height, IQ) of the best Y (e.g. basketball player, wealthiest) for a given correlation."

A snippet you might find interesting. True or False: Half of All 7-Footers are in the NBA

An actual accounting of 7-footers, domestic or global, does not exist in any reliable form. National surveys by the Center for Disease Control list no head count or percentile at that height. (Only 5% of adult American males are 6’3″ or taller.)…

The curve shaped by the CDC’s available statistics, however, does allow one to estimate the number of American men between the ages of 20 and 40 who are 7 feet or taller: fewer than 70 in all. Which indicates, by further extrapolation, that while the probability of, say, an American between 6’6″ and 6’8″ being an NBA player today stands at a mere 0.07%, it’s a staggering 17% for someone 7 feet or taller.

(new site) Economics Institute said...

Verbal is hard. You make a g loaded test and you're gonna have a skewed distribution with too many high scorers, similar to how math score on SAT/GREs are always higher than verbal for most of the population. Kinda makes you think twice before dismissing non-STEM as being 'dumb'.

Richard Seiter said...

Thanks! I started work on a simulation of this problem. The initial version which just shows a plot of a population of 10,000 is at http://rpubs.com/rseiter/23845


10k population only gets near 4 SD so I'm a little worried about finding a practical and effective sample size. If anyone has the probability/statistics chops to solve this analytically (I don't) that is probably the way to go.

Pumpkinperson.com said...

It would be interesting to calculate likelihoods for the X (e.g. height, IQ) of the best Y (e.g. basketball player, wealthiest) for a given correlation. I wonder what the MLE would be and how it would compare to Bill Gates' IQ (since we can actually make a decent estimate given his SAT score). It would probably be even better to look at the top n instead.

I did do exactly that except to predict Oprah's IQ (not Bill Gates). Oprah's FAR more fascinating than Bill Gates, because while Gates is an extreme outlier on income, Oprah's an extreme outlier on BOTH income AND brain size, thus a Promethean suggested I use MULTIPLE REGRESSION to estimate her IQ:

http://brainsize.wordpress.com/2014/07/08/multiple-regression-predicts-oprahs-iq-using-income-brain-size/

Income and brain size are the two most Darwinian correlates of intelligence and both correlate 0.4 with IQ so for every 1 SD increase in either trait, you expect IQ to increase by 0.4 SD. But to find someone who was so extreme on BOTH traits is really interesting because both contribute very independently to the expected IQ (unlike income and education which are so intercorrelated themselves).

It was like trying to predict the height of the world's best basketball player, knowing they also wear the world's biggest shoe (TWO correlates of height)

Richard Seiter said...

That is interesting, but not the same as what I was proposing. Oprah is definitely an intriguing case--especially so for me since I grew up watching the Baltimore news during the time she came onto the scene there. It is amazing how far she has come since then.

Pumpkinperson.com said...

Well let's say the average Terman gifted kid is 2.5 SD above average in IQ. Given the 0.65 correlation between IQ and academic success they should by 2.5 SD(0.65) = 1.63 SD above average academically. So roughly the top 5% on average. You're right, that would be more than a college degree, though less than a PhD which is the top 1%. And you also have to factor in that the Terman study was back in the days when a college degree put you in an exclusive club.

Pumpkinperson.com said...

What you're proposing (if I understand you) sounds like it's just the standard error around a prediction. So given a 0.4 correlation between IQ and income, Gates' expected IQ would be 134 with an SE of 13.75, so there would be a 95% chance his IQ is between 107 and 162. If I'm totally missing your point, please forgive me. :-)

Richard Seiter said...

I did a simulation as a sanity check. Assuming a correlation of 0.65 of outcome Y with independent variable X, the most likely value of X for maximum Y was between 2.5 and 3 SD. Looking at probability of individual success the simulation (unsurprisingly) agrees with Steve's assertion (the probability keeps increasing past 5 SD (at that point I think sampling error takes over).

Details (including the R code) at http://rpubs.com/rseiter/23845

Richard Seiter said...

I'm proposing something rather different. See http://rpubs.com/rseiter/23845 It's similar to your second idea.

Pumpkinperson.com said...

In that link you write "calculate the likelihood that a given X will have the largest Y". If you're asking, what's the likelihood of the richest (self-made) person in America being the smartest person in America, then the likelihood would be about one in 10,000.

Pincher Martin said...

I think there are many better ways to assess the academic success of the top one percent than by using a rough grouping where educational attainment is divided into just five groups - high school dropouts, high school grads, college grads, gradate degree holders, and PhDs - and a PhD in educational leadership from Fresno State is assumed to be more impressive than, say, a bachelor's degree in mathematics from Princeton.

Terman, for example, looked at educational aspirations, accelerated education (i.e., how quickly did the subjects graduate), the quality of education, the subjects studied, grades, etc. By all those measures, his test subjects outperformed not just the general population, but their college-educated peers. In some cases, they did so by large margins.

Richard Seiter said...

Perhaps, though the point of simulation is to be able to give numbers with at least a little justification. The question I am asking would be better phrased as "what is the likelihood of the richest (I'm not sure explicitly saying self made matters here, is anyone in the top 10 inherited wealth?) person being >3 SD in IQ?" I think likelihood by X is a better way to express it though. And the chance per person by X is quite interesting--and the clearest way to argue against the "over a threshold is irrelevant" meme IMHO.

panjoomby said...

well explained!
but i was hoping to see the word "heteroskedasticity":)

Pumpkinperson.com said...

Well assuming there are about 242 million American adults in America, the richest American can be viewed as having a (cumulative) income normalized Z score of 5.73 (one in 242 million level). Given the 0.4 correlation between IQ and income, his expected IQ would have a Z score of 5.73(0.4) = 2.29. Given a 0.4 correlation, the standard error of that estimate would be 0.92. So an IQ > 3 SD would be 0.77 SE above the predicted value, so the odds of that occurring would be about 21%. So, yeah, the higher the income, the greater the probability that one would be above a 3 SD IQ.

reservoir_dogs said...

IQ is an imperfect measurement of one of the dimensions needed for "success". Aside from luck, I will bet that there are a number of genetic factors for every kind of success. When a farmer looks for a good dairy cow, he does not just look for a cow with the biggest udder. In fact, he looks for the cow that gives the most milk. The udder size is a by-product of this selection.


I think if we are to look for someone with some sort of success, we should do just that, and IQ is a natural by-product of it, recognizing that there are number of other factors, many of which are genetic but only some what correlated to IQ.

steve hsu said...

? The fact that the V ceiling on SAT/GRE is higher than for M is the choice of the test makers. They could have done it the other way as well.

steve hsu said...

IIRC two of the experimental physicists in her sample had pretty mediocre (~120) V scores. One of them could have been Alvarez. The V mean for all the scientists was very high - 166!


SMPY is a good modern update of Terman.

Richard Seiter said...

I think you recall correctly about Roe. Pp 162-3 has the relevant table and text but gives raw scores. A version of her chapter 12 tables (including that one) converted to IQs appears in http://psychology.ucdavis.edu/faculty_sites/simonton/greatpsychologistspiiich6.pdf (I think this is worth a look, it appears to be a chapter from Simonton's Great Psychologists book). Also note Roe's comment about the lowest scoring experimental physicist may not have done justice to him?self.

Have you seen Anne Roe's 1953 monograph "A Psychological study of eminent psychologists and anthropologists, and as a comparison with biological and physical scientists"? Worldcat says MSU has a copy. There is an excerpt at http://www.physics.umd.edu/~einstein/HONR228K/Roe.pdf but I suspect the monograph is more likely to have data.

Dutton and Lynn's 2014 "Intelligence and Religious and Political Differences Among Members of the U.S. Academic Elite" http://www.religjournal.com/pdf/ijrr10001.pdf (published in a religious journal?!) mentions Gibson and Light (1967) http://psycnet.apa.org/psycinfo/1967-11916-001 as a study similar to Roe's and has some other interesting bits (e.g. the Big 5 in academics).



Good point about the SMPY. The initial 2188 in top 1% cohort is a good basis for comparisons. Your quartile graphic of the top 1% is a great example. Did they break out the top 1% any further than quartiles? 1% to 0.25% is about 2.5 to 3 SD. The 0.01 % cohort (~3.9SD cutoff, 501 people) does this, but the within cohort comparisons seem most helpful. I guess the problem is there would only be ~20 in their first cohort of 2188 above ~4SD. Still, it would be interesting to see odds ratios broken out for 3, 3.5, 4, + SD groups.

Pumpkinperson.com said...

I'm still quite baffled by that 166 mean V score Roe obtained. IQ is supposed to be normally distributed so 166 would imply one in 185,000 level ability. It would be somewhat equivalent to scoring near the ceiling of the verbal SAT (new scale) at age 13! It's hard to imagine the average (verbal) IQ of any group of people, no matter how accomplished, being that high given the imperfect correlation between ability and achievement. I speculate that Roe did not have any extremely gifted people in her test norming sample and thus was forced to assign extreme IQ scores through linear extrapolation which works on an interval scale, but I suspect Roe used ordinal scales.


Thus I think the more meaningful data in Roe's study is the lowest score. If the least brilliant scientist (of the 64) had an IQ of 121 (and assuming he/she wasn't an outlier), then IQ 121 should be 2.13 SD below the mean of all the scientists. If we assume the scientists had an SD of 13.5 (similar to college grads) then the average eminent scientist should have a (verbal) IQ of 2.13(13.5) + 121 = 150.


As I mentioned above, one in a million level academic achievers should AVERAGE IQ's around the one in 2000 level given the 0.65 correlation between IQ and scholastic success, so an IQ of 150 would make perfect sense.

steve hsu said...

I have two posts on the blog about Roe's results and the identities of the scientists she studied.

steve hsu said...

I think what we can safely conclude from Roe is that her top scientists were unusually smart compared to average scientists.

In the book, she says she normed her high ceiling tests on a population of Columbia Teacher's College graduate students, who also took standard IQ tests. Once she knows the SD in that group she can use it to convert high end raw scores to z scores. Perhaps the student SD is smaller than the general population SD and she failed to adjust for that. But she had their scores on well normed standard tests, so she could and should have done so. She didn't do the psychometric work herself -- she worked with a specialist at Columbia.

Also, there is probably some deviation from normality at the high end.

Pumpkinperson.com said...

I agree. The eminent scientists were way smarter than average scientists, and the test was probably properly normed, but the distribution was probably non-Gaussian at the high end, either because the item difficulty increased in a non-linear way or because high verbal ability itself departs from the Gaussian curve.

Richard Seiter said...

Right. You might recall I was the reader who pointed you to the list of names (prompting the second of those posts ;-). My thought was the monograph https://www.worldcat.org/title/psychological-study-of-eminent-psychologists-and-anthropologists-and-a-comparison-with-biological-and-physical-scientists/oclc/1441032 is different (note the different page counts and titles) from the book https://www.worldcat.org/title/making-of-a-scientist/oclc/526273 (could they have the same content with a 4x different page count? I guess so if one was the paperback format I have and the other was dense text on larger pages.)

If I missed mention of the monograph (with the title I gave) in your other posts or am somehow otherwise mistaken I apologize, but I think the monograph (if different) might be more likely to contain hard data, references, and figures than the book. I don't have access to a library with it or I would chase it down to check.

The Gibson and Light article looks interesting, but at 5 pages long it seems unlikely to have that much content. Check out the subtitle (was it an originator of the meme? ;-) https://www.worldcat.org/title/intelligence-among-university-scientists-studies-on-the-deviation-iq-of-scientists-at-the-university-of-cambridge-show-that-there-is-a-considerable-variation-among-scientists-however-iq-seems-not-to-be-related-to-success-as-a-scientist-provide-that-it-is-greater-than-a-certain-threshold-which-varies-according-to-the-particular-scientific-discipline/oclc/25881123

It is worth noting that that paper looked at scientists in general (at Cambridge) without selecting for "eminence".

Richard Seiter said...

Regarding the low V scoring scientists, Roe makes a statement (pp 162-163) which leads me to think two of the scores (a biologist and experimental physicist) should be treated as outliers (she says she noted something like that even before scoring the tests - "did not do justice to themselves").


It might be worth checking what "scholastic success" metric is used in your correlation. Scientific eminence is different from grades. Also, I'm not sure how much I trust a linear correlation pushed out to a >4 SD tail. Correlations are handy, but they do have limitations as analytical tools IMHO. (but they do very nicely meet my criteria of justifiable numbers so please keep the analysis coming, I appreciate that ;-)


If you have the inclination Roe's methodology for assigning IQs might be worth a closer look. Steve summarized it, and it looks to me like she made a credible effort to "do it right." Elsewhere in this thread I mentioned the 1967 Gibson and Light study which used the WAIS.

Emil Kirkegaard said...

Nice. I already wondered how to create two randomly generated normal distributions that are correlated.

Emil Kirkegaard said...

The average Termite was 152, sd=15.

Richard Seiter said...

Thanks! Google and Stack Overflow are a great combination for answering questions like that (I had to answer that question to do the simulation).
I updated the simulation to include a look at the top 0.1% of Y as well as the largest Y (and comparing the two). Interesting (though perhaps not surprising) to see how the curves shift. The way the top 0.1% density curve peaks near 2SD makes clear how low the X level can be for the high Y's most observed (though I wonder about the actual threshold effect in science, e.g. you need a certain IQ to make it at all, if that is so how would it affect this analysis?) while the per-person curves show how increasing returns to higher X can be consistent with the observed density.

Richard Seiter said...

Thanks for posting the numbers. Those seemed high to me so I did a simulation of what a truncated normal distribution would look like. See http://rpubs.com/rseiter/24130
Truncating at 135 I got a mean of 140 and sd of 4.57 for the sample.


Any thoughts as to why there are would be such a dramatic difference? Are the tails of the IQ distribution that fat? Any ideas what distribution might give a better fit to the Terman data? I tried lognormal and it gave mean 142.5 and sd 7.05 so still not nearly high enough.

stevesailer said...

Reasoning statistically is just something that's culturally new to us. The concept of regression toward the mean wasn't worked out until 1885, 198 years after Newton's big book.

stevesailer said...

That's about, very roughly, where Michael Jordan fell on the height distribution: about 2.5 to 3.0 standard deviations above the mean height. LeBron James is another s.d. out to the right.

Pumpkinperson.com said...

Yes, there are many ways you can measure scholastic success. Elementary school grades in children and years of education among adults both correlate 0.65 with IQ so I use 0.65 as a good estimate for scholastic success in general (in the total population, correlations are much smaller in restricted samples like grad students). So if there were a perfect correlation between IQ and scholastic success, we'd expect the dumbest academic Nobel prize winner to have an IQ of 171 (one in a million on a Gaussian scale) and the average to be maybe 175? (correct me if I'm wrong) But since the correlation is 0.65, their average expected IQ would be 0.65(175 - 100) = 149. The reason I'm so suspicious of Roe's average IQ of 166 is it would imply a 0.88 correlation between IQ and academic success (actually maybe higher since Roe's scientists were below the Nobel prize level) which is higher than IQ tests correlate with each other


But I understand that such calculations have limitations. There are different types of academic success and it's simplistic to just use 0.65 as a one size fits all correlation for criteria as diverse as elementary school grades, years of education, and scientific eminence, and it's also simplistic to assume correlations observed in the general population can be extrapolated to predict one in a million achievements. But occam's razor suggests that we start with simple assumptions and only abandon them when they become untenable I also just find linear relationships aesthetically pleasing and am biased in favor of them which is probably how I got interested in IQ in the first place.

steve hsu said...

Oops, sorry I didn't realize the monograph might be different from the book. Might be worth looking into.

I've read the Cambridge paper and it's not very enlightening. The scores seem very low to me.

Pumpkinperson.com said...

The average Termite did indeed score in the 150s, but ratio IQ's used at the time generally had SDs greater than 15 and were non-normally distributed at the extremes because some kids have cognitive growth spurts causing their mental age to leap to twice their chronological age, thus their NORMALIZED average IQ was likely well below the 3 SD level

Cornelius said...

V-IQ tends to increase somewhat with age:

http://qph.is.quoracdn.net/main-qimg-fd832e64253819688a4eaaaf25eac62b?convert_to_webp=true


Let's take that graph at face value and assume the test was normalized in such a way that average 20-somethings would score 100 for V-IQ. At the same age, the theoretical physicists in the group would have scored about 164 V-IQ. That sounds about right to me.


Nobel Prize winners are not noticeably smarter than average faculty at top schools such as MIT and Harvard. When I think of theoretical physicists at that level, a verbal IQ at the 1:100,000 level seems about right. 1:10,000 is probably too low, but it could be a lower bound for those eminent theoretical physicists whose main strength is non-spatial reasoning.

Richard Seiter said...

Agreed about the value of simple (here and usually, linear) models -- and well put in your last paragraph. Your calculations agree reasonably with my simulation, if anything they seem perhaps to give a higher estimate. With a 0.65 correlation I see the center of the X density a little over 2 SD for the best and about 3SD for the top 0.1%. I would expect a Nobel prize to fall between the two. Given all the assumptions I'm not sure I trust the simulations any more than your linear models.

Did you see the list of Roe's scientists linked in Steve's earlier post? http://www.amphilsoc.org/mole/view?docId=ead/Mss.B.R621-ead.xml



One other comment (perhaps one of the practicing scientists here can add to this). How exactly

Richard Seiter said...

Can you elaborate on this? A reference expanding on this point would be appreciated.

Richard Seiter said...

Interesting graph. I went looking for it and found http://www.sibille.pitt.edu/Publications/2011_Glorioso_ProgressNeurobiology.pdf
which looks at nervous system aging and disease. Figure 3 emphasizes the heterogeneity of the population underlying the average curves.

AG said...

Concur with you.
We all noticed that some kids who were so smart in our grade school yet falled behind in high school years. Kids too young yield unreliable intellectual information. People might have very different mental maturation rate just like that of height.

Blog Archive

Labels