Pinker on Gladwell

Ouch!

Thanks to a reader for pointing out this Steve Pinker review of Malcolm Gladwell's latest collection in the Sunday Times. I had more or less stopped reading stuff on Gladwell, as the uncritical acceptance of many of his claims is just too depressing a reminder of the mediocrity of our commentariat.

An eclectic essayist is necessarily a dilettante, which is not in itself a bad thing. But Gladwell frequently holds forth about statistics and psychology, and his lack of technical grounding in these subjects can be jarring. He provides misleading definitions of “homology,” “saggital plane” and “power law” and quotes an expert speaking about an “igon value” (that’s eigenvalue, a basic concept in linear algebra). In the spirit of Gladwell, who likes to give portentous names to his aperçus, I will call this the Igon Value Problem: when a writer’s education on a topic consists in interviewing an expert, he is apt to offer generalizations that are banal, obtuse or flat wrong.

...

The common thread in Gladwell’s writing is a kind of populism, which seeks to undermine the ideals of talent, intelligence and analytical prowess in favor of luck, opportunity, experience and intuition. For an apolitical writer like Gladwell, this has the advantage of appealing both to the Horatio Alger right and to the egalitarian left. Unfortunately he wildly overstates his empirical case. It is simply not true that a quarter­back’s rank in the draft is uncorrelated with his success in the pros, that cognitive skills don’t predict a teacher’s effectiveness, that intelligence scores are poorly related to job performance or (the major claim in “Outliers”) that above a minimum I.Q. of 120, higher intelligence does not bring greater intellectual achievements.

My opinion of Malcolm Gladwell was expressed here:

Malcolm Gladwell shows exquisite taste in the subjects he writes and talks about -- he has a nose for great topics. I just wish his logical and analytical capabilities were better ... My feeling is that Gladwell's work appeals most to people who can't quite understand what he is talking about.

What Pinker refers to as the major claim of Outliers: IQ above 120 doesn't matter, is easily shown to be false. Randomly selected eminent scientists have IQs much higher than 120 and also much higher than the average science PhD (120-130); math ability within the top percentile measured in childhood is predictive of future success in science and engineering; advanced education and a challenging career do not enhance adult IQs relative to childhood IQ.

So, accomplished scientists tend to have high IQs, and their IQs were already high before they became scientists -- the causality is clear. 10,000 hours of practice may be necessary but is certainly not sufficient to become a world class expert.

I recently remarked to a friend that many aspects of psychometrics which were well established by the 1950s now seem to have been completely forgotten due to political correctness. This leads to the jarring observation that recent social science articles (the kind that Gladwell is likely to cover) are sometimes completely wrong headed (even, contradicted by existing data of which the authors are unaware) whereas many 50 year old articles are clearly reasoned and correct. The data I cite in the links above comes from the Roe study of eminent scientists and the Terman longitudinal study of gifted individuals, both of which were conducted long ago, and the SMPY longitudinal study of mathematically precocious youth, which is ongoing. I've interacted with many social scientists whose worldview is inconsistent with the established results of these studies, of which they are unaware.

Gladwell and genius

Malcolm Gladwell shows exquisite taste in the subjects he writes and talks about -- he has a nose for great topics. I just wish his logical and analytical capabilities were better (see also here). This talk at the New Yorker's recent Genius 2012 conference is entertaining, but I disagree completely with his conclusion. Ribet, Wiles, Taniyama and Shimura are probably the real geniuses, not Michael Ventris, the guy who decoded Linear B. (Gladwell also can't seem to remember that it's the Taniyama-Shimura conjecture, not Tanimara. He says it incorrectly about 10 times.) My feeling is that Gladwell's work appeals most to people who can't quite understand what he is talking about.

Gladwell is confused about the exact topic discussed in James Gleick's book Genius. In a field where sampling of talents is sparse (e.g., decoding ancient codexes) you might find one giant (even an amateur like Michael Ventris) towering above the others, able to do things others cannot. In a well-developed, highly competitive field like modern mathematics, all the top players are "geniuses" in some sense (rare talents, one in a million), even though they don't stand out very much from each other. In Gleick's book, Feynman, discussing how long it might have taken to develop general relativity had Einstein not done it, says "We are not that much smarter than each other"!

To put it simply, if I sample sparsely from a Gaussian distribution, I might find a super-outlier in the resulting set. If I sample densely and have a high minimum cutoff for acceptable points, I will end up with a set entirely composed of outliers, but who do not stand out much from each other. Every guard in the NBA is an athletic freak of nature, even though they are relatively evenly matched when playing against each other.

To counteract the intelligence-damping effect of Gladwell's talk, I suggest this podcast interview with Nassim Taleb, about his new book The Black Swan. Warning: may be psychologically damaging to people who fool themselves and others about their ability to predict the behavior of nonlinear systems.

Kakutani on Gladwell

Michiku Kakutani of the Times reviews Malcolm Gladwell's new book Outliers. She finds it poorly reasoned -- my usual complaint about Gladwell's work.

Much of what Mr. Gladwell has to say about superstars is little more than common sense: that talent alone is not enough to ensure success, that opportunity, hard work, timing and luck play important roles as well. The problem is that he then tries to extrapolate these observations into broader hypotheses about success. These hypotheses not only rely heavily on suggestion and innuendo, but they also pivot deceptively around various anecdotes and studies that are selective in the extreme: the reader has no idea how representative such examples are, or how reliable — or dated — any particular study might be.

Gladwell highlights the claim of psychologist Anders Ericsson, that effort dominates ability (the 10,000 hours of practice thesis). My opinion on this can be found here, deep in the comments. The evidence is pretty strong in the case of science that native cognitive ability is a prerequisite for success. Practice (effort) is necessary also, but neither alone are sufficient.

...that quote sounds like it could be from Anders Ericsson's research on expertise. I disagree with his conclusions. His studies only show that effortful practice (about 10 years worth) is typically required to reach the highest level of capability. But he then confuses the logic and asserts that practice alone is *sufficient*, when in fact it is only necessary. You need raw ability *and* lengthy practice to reach expertise.

Of course it is appealing for most people to think that Ericsson's model is correct and that effort is all that is required to produce capability, but this claim is very controversial in the psychology community, and I think implausible to anyone who has been around gifted kids/adults.

The Roe study, combined with other studies showing the age stability of IQ (certainly once adulthood is reached), also serves to refute Ericsson. There's clearly some measurable quality, usually present already at an early age, that is advantageous for intellectual achievement. Most people don't have it.

Anders is refuted quite well in papers by leading psychologists like Sternberg (Yale) and in Eysenck's book Genius.

By the way, also contra Ericsson, there are many credible examples of supreme raw talent that didn't require development through 10 years of practice (e.g., Mozart).

Social Animals

The WSJ reviews David Brooks' new book The Social Animal. I discussed an excerpt which appeared in the New Yorker in this earlier post. David Brooks is in danger of becoming the new Malcolm Gladwell: watch out for sloppy thinking, confirmation bias, and pandering to populist sentiments :-(

WSJ: ... In "The Social Animal," Mr. Brooks avoids the challenge of imitating the master by reviving an older form: the 18th-century didactic narrative, epitomized by Rousseau's "Emile." Rather than dig up true stories, he writes one from scratch, and it's not bad. His main characters are Harold and his wife and business partner, Erica. The plot extends all the way from the day Harold's parents meet to the day Harold dies, yet it takes place in a perpetual present-day America.

The chapters are at once chronological and thematic, tackling an array of topics, like learning (a high-school teacher sparks Harold's intellectual curiosity), social norms (Erica has youthful troubles with authority figures), morality (Erica sleeps with a business leader whom she admires), politics (Harold and Erica work for a presidential candidate), and retirement (they move to Aspen and start a tour business).

[See earlier post for more on Harold, Erica, and their Aspen encounter with the Caltech neuroscientist.]

Mr. Brooks is at his best as a social observer, documenting the changing patterns of contemporary life. He showed that talent in "Bobos in Paradise" (2000)—a modern classic that deconstructed every signifier of millennial upper-class life, from Restoration Hardware catalogs to New York Times wedding announcements. Several passages in "The Social Animal" made me think that Mr. Brooks had been glancing through my own windows or fishing from the banks of my own stream of consciousness.

As in "Bobos," he shows genius in sketching archetypes and coining phrases. Here we learn of the "composure class" (who earn their money "by climbing the meritocratic ladder of success"), "sanctimommies" (mothers who critique their peers' parenting skills) and "extracurricular sluts" (kids who participate in too many organized after-school activities).

... If this story is meant to illustrate a broader point, it must be that "cognitive intelligence" and "emotional intelligence" have an inverse relationship: The brilliant are more likely than the average to be socially awkward. But this is nothing but the stereotype that forms the premise of "The Big Bang Theory" and other pop-culture narratives. In reality, tests of emotional intelligence correlate positively with IQ tests.

But Mr. Brooks makes an even bigger claim: Not only is intelligence not connected to social skill; it's not connected to much of anything. "Once you get past some pretty obvious correlations (smart people make better mathematicians), there is a very loose relationship between IQ and life outcomes." This conclusion misstates the science. To reach it Mr. Brooks has to commit a variety of statistical errors and tiptoe through a minefield of contradictory evidence.

First, he notes that studies selecting a group of high-IQ kids and tracking them over decades have failed to identify future Pulitzer or MacArthur winners. He also cites Mr. Gladwell's "Outliers" to note that American science Nobelists "did not mostly go to Harvard and MIT."

[Never rely on Gladwell in statistical matters. Instead, see here and here.]

The problems here include the tiny sample sizes, the shaky assumption that prize juries (and elite universities) make decisions based only on merit, and the focus on the tails of a distribution (here, the highest extremes of intelligence and academic achievement), which is the method guaranteed to tell you the least about the characteristics that matter across the whole range of human ability. To dismiss IQ testing as invalid because it can't pick out the minuscule minority that will attain world-wide fame is to confuse a positive correlation with a perfect one. Only oracles have perfect records of prophecy, and surely no one desires a world in which IQ tests are that good.

[Using a quick IQ screen at age 17 probably allows you to identify future scientific leaders with 100 times better accuracy than chance, and future business leaders with 5-10 times better accuracy than chance. This follows immediately from (soft) 99+ and 90th percentile IQ thresholds. That is, the fraction of people in each of the two categories that is below the given threshold is very small. See links above.]

Next Mr. Brooks cites a study that found "no correlation between accumulating large wealth and high IQ." But in fact, IQ (measured in high school) was almost the best predictor of wealth (measured in early middle age) in this study's own data set. Its author did show that if you consider only people matched in education, income and other variables, then intelligence is unrelated to wealth. But this is hardly a surprise, since education and income themselves both correlate with IQ. In other words, if you look only at, say, high earners with graduate degrees, you will see that the smartest ones are no richer than the dumbest ones, but you will have learned nothing about the true relationship between intelligence and success.

Similarly, Mr. Brooks describes research showing that, during the dot-com bubble, the average investor in many mutual funds lost money by trading in and out too much, when the investor could have made money by just sitting tight. Such people acted "self-destructively because of their excessive faith in their intelligence," he says. Surely intelligent people can do stupid things. But where is the evidence that intelligence or intellectual arrogance was correlated with bad decisions, let alone caused them? The mutual-fund study said nothing about whether more intelligent investors were more overconfident, or traded more, than less intelligent investors.

In fact, studies that do measure both intelligence and economic outcomes find positive relationships. When you consider that IQ can be measured well in half an hour, these effects are surprisingly large: A random person with above-average intelligence has about a two-thirds chance of being above average in income, compared with a one-third chance for a random person of below-average intelligence.

The research that Mr. Brooks minimizes or ignores does not, of course, prove that intelligence is the only relevant trait for success. A host of "noncognitive" skills, many of which Mr. Brooks explains well, are undoubtedly important. But there is no need to tear down intelligence in order to build up the rest.

Even if differences in intelligence explain 25% of the differences among people in how well they perform at work (a much better estimate than the low-ball 4% cited by Mr. Brooks), there is still three times as much territory left to be mapped out. Surely that's plenty of space for researchers to investigate the role of social acumen, mindset, culture, self-control and much else. A thousand flowers can bloom.

At one point, Mr. Brooks veers into odd territory. Economists and political scientists, he writes, assume that "human beings are pretty much the same. . . . This assumption makes social science a science. If behavior is not governed by immutable laws and regularities," he writes, "then quantitative models become impossible. The discipline loses its predictive value." Perhaps this is an allusion to traditional theories in these fields that neglected differences between people for the sake of mathematical simplicity. But the claim is badly out of date. In fact, human homogeneity is neither an assumption in today's social science nor a requirement of the scientific method. It's ironic that Mr. Brooks neglects the abundant evidence for a key source of differences among people—their genes—which influence their personalities, decisions and achievements.

Mr. Brooks would have also done well to suspend his disbelief when his sources talked about brain research. "When stock traders experience a series of good days, the dopamine released into their brains creates a surge of overconfidence," he writes. He cites only a blog post for this claim. Most of the sources in "The Social Animal" are secondary ones, usually books for a general audience. Without information about the primary sources—like journal articles that report methods and results—it is hard to trust that the science is being communicated and interpreted fairly. In the case of the traders hopped up on dopamine, the story sounds good but the necessary experiments just haven't been done.

In "The Social Animal" Mr. Brooks surveys a stunning amount of research and cleverly connects it to everyday experience. The lessons he draws are often insightful, but they are not reliably correct. Perhaps experiencing his own surges of dopamine and overconfidence, he too often abandons his stance of "epistemological modesty" and instead peddles frothy notions that probably won't last long. But in observing the broader trends of social science—and of contemporary life—he gets a lot right. His own achievement here signals a plateau in the market for social science, not a peak.

If you're so smart, why aren't you rich?

Does being smart help you become rich?

Folksy Warren Buffett once said that an investor with IQ of 150 should sell 30 points to someone else, as anything above 120 is unnecessary.

Consider the following simple model (we can call it the "Igon Model" in honor of Malcolm Gladwell):

Igon Model: IQ correlates positively with wealth, but the effect goes away for IQ > 120. IQ above 120 provides no advantage, relative to IQ=120, for acquiring wealth.

Were this model to be true, one would expect with overwhelming probability to find that the vast majority of rich people have IQ around 120, but not much higher. This is because IQ is normally distributed: as you go further out the tail the population decreases exponentially. To be specific, IQ = 120 corresponds to the 90th percentile, whereas IQ = 135 is 99th percentile (i.e., only 1 in 10 people with IQ > 120 have IQ > 135) and IQ = 145 is 99.9th percentile (i.e., only 1 in 100 people with IQ > 120 have IQ > 145).

Now let's look at the 2009 Forbes list of richest people in the world:

1 William Gates III 53 40.0 United States
2 Warren Buffett 78 37.0 United States
3 Carlos Slim Helu 69 35.0 Mexico

If the Igon Model were correct, we would not expect to find this list dominated by people with IQ much higher than 120. But in fact we do. Note these three made their money in different ways: Gates founded a software company, Buffett is primarily an investor, and Carlos Slim is an oligarch ;-)

Bill Gates scored 1580 on the pre-1995 SAT. His IQ is clearly >> 145 and possibly as high as 160 or so.

Warren Buffett graduated high school at 16 ranked in the top 5 percent of his class despite devoting substantial effort to entrepreneurial activities. Most people who know him well refer to him as brilliant, that folksy quote above notwithstanding. I would suggest the evidence is strong that his IQ is above 135, perhaps higher than 145.

Carlos Slim studied engineering and taught linear programming while still an undergraduate at UNAM, the top university in Mexico. He reportedly discovered the use of compound interest at age 10. I would suggest his IQ is also at least 135.

So it would appear that the three richest men in the world all have IQs that are higher than 90 percent or even 99 percent of the > 120 IQ population. (Relative to the general population they are all likely in the 99th or even 99.9th percentile.) The probability of this happening in the Igon Model is less than 1 in 1000.

[Here's a basketball analogy: the analogous Igon Model for basketball would say height over 6ft2 (90th percentile) doesn't increase likelihood of success in basketball. Suppose we find the 3 top players in the world are 7ft (Shaq/Gates), 6ft8 (LeBron/Buffett) and 6ft6 (Kobe/Slim). That strongly disfavors the model, as a random draw of 3 people from the set of people over 6ft2 in height has almost zero probability of producing the 3 heights we found.]

Note to angry Gladwell egalitarians: don't take this analysis too seriously :-) It's really an example of "Igon analysis" in the spirit of MG!

There are many factors aside from intelligence that impact success in business or investing. See here for a discussion by money manager and investment theorist William Bernstein, which is very similar to what Buffett has said on various occasions. If you carefully study biographies of the three men listed above, what really stands out (aside from high mental ability) is their determination, drive and fascination with material success beginning at a young age. See also: success vs ability and creators and rulers.

What about the broader population? It's well established that graduates of elite universities earn more than graduates of less selective schools. But, interestingly, controlling for SAT score (IQ) largely eliminates the differential. I wonder why? (See also here for UT Austin data on earnings variation with SAT and major.)

Among the patent trolls

This American Life makes the case that Intellectual Ventures are patent trolls cum extortionists. Thanks to a reader for sending me the link. See also this earlier post which examines Malcolm Gladwell's New Yorker article on Nathan Myhrvold and IV. (Gladwell, as usual, gets it all wrong.)

Transcript: ... Alex: We told Intellectual Ventures that Chris Sacca compared their business to a mafia shakedown and in an e-mail, Peter Detkin called that ridiculous and offensive. He then reiterated some of the arguments you’ve heard about how IV protects inventors and went on to say, “We’re a disruptive company that’s providing a way for patent-holders to recognize value.” (By “recognize value,” he means “make money.”) “That wasn’t available before we came on the scene, and we are making a big impact on the market. That obviously makes people uncomfortable. But no amount of name-calling changes the fact that ideas have value.”

Laura: True enough, but lately it seems like a lot of butcher shops have been burning. As we were reporting this story, more and more Intellectual Ventures patents started showing up in the hands of companies like Oasis, companies without employees or operations, who were formed for the purpose of filing lawsuits. They’re known as nonpracticing entities or NPE’s.


... that’s based just on the math of IV’s business model. In order to purchase its 35,000 patents, Intellectual Ventures got money from investors. A lot of money. More than $5 billion dollars.

Laura: And a lot of these investors are venture capitalists who expect very high returns. These are people who are looking for the next Google, the next Apple. People who want to get back many times what they put in. Since its founding in 2000, Intellectual Ventures has generated $2 billion dollars in revenue. But to keep its investors happy, over the next 10 years, says Tom Ewing, they’re going to have to do a lot better than that.

Ewing: So if you calculate this out, that means that over say a 10-year period they’re going to need to collect about $35 billion dollars in licensing revenue, in order for them to be successful among the people who they’re trying to compare themselves with. IV seems to have signed a number of deals. If the stream of deals they’re signing doesn’t increase significantly, then I would imagine they will be forced to file more litigations in order to achieve their revenue targets.

Laura: Tom’s prediction already seems to be coming true. Earlier this month, Intellectual Ventures itself filed a patent infringement suit in federal court against several companies it claimed were infringing some patents it owns.

Laura: In early July, the bankrupt tech company Nortel put its 6,000 patents up for auction as part of a liquidation. A bidding war broke out between the Silicon Valley powerhouses. Google said in press accounts that it wanted the patents purely to defend itself against lawsuits and it was willing to spend over $3 billion dollars to get them. But that wasn’t enough. The portfolio eventually sold to Apple and a strange consortium of other tech companies, including Apple competitor Microsoft.* The price tag? 4.5 billion dollars. Five times the opening bid. More than double what most people were expecting. The largest patent auction in history.

Alex: Think of that — 4.5 billion dollars on patents that these companies almost certainly don’t want for their technical secrets. That 4.5 billion dollars won’t build anything new, won’t bring new products to the shelves, won’t open up new factories that can hire people who need jobs. That’s 4.5 billion dollars that adds to the price of every product these companies sell you — 4.5 billion dollars essentially wasted, buying arms for an ongoing patent war. The big companies, Google, Apple, Microsoft, will probably survive this war. The likely casualties, the companies out there now that no one’s ever heard of that could one day take their place.

Teaching effectiveness

The two figures below (click for larger versions) are taken from the Brookings report by Gordon, Kane and Staiger: Identifying Effective Teachers Using Performance on the Job. The report has received a lot of attention recently thanks to Malcolm Gladwell's New Yorker article. Both are worth a look if you are interested in education. The top figure shows that certification has no impact on teaching effectiveness. The second shows that effectiveness measured in the years 1 and 2 is predictive of effectiveness in the subsequent year. In this case effectiveness is defined by the average change in percentile ranking of students in the teacher's class. Good teachers help their students to improve their mastery, hence percentile ranking, relative to the average student studying the same material.






It's obvious to me that there is gigantic variation in effectiveness among teachers. Gladwell emphasizes how difficult it is to evaluate teaching capability in initial hiring, and how the single most important impact on overall school effectiveness is due to individual teachers (he also makes the analogy to scouting college QBs for pro football -- it's very hard to predict NFL performance based on college performance). The Brookings paper has many policy suggestions, but the basic idea is that if we were disciplined and data-driven we could easily determine which teachers are good and which ones are not.

New Yorker: ...One of the most important tools in contemporary educational research is “value added” analysis. It uses standardized test scores to look at how much the academic performance of students in a given teacher’s classroom changes between the beginning and the end of the school year. Suppose that Mrs. Brown and Mr. Smith both teach a classroom of third graders who score at the fiftieth percentile on math and reading tests on the first day of school, in September. When the students are retested, in June, Mrs. Brown’s class scores at the seventieth percentile, while Mr. Smith’s students have fallen to the fortieth percentile. That change in the students’ rankings, value-added theory says, is a meaningful indicator of how much more effective Mrs. Brown is as a teacher than Mr. Smith.

It’s only a crude measure, of course. A teacher is not solely responsible for how much is learned in a classroom, and not everything of value that a teacher imparts to his or her students can be captured on a standardized test.

Nonetheless, if you follow Brown and Smith for three or four years, their effect on their students’ test scores starts to become predictable: with enough data, it is possible to identify who the very good teachers are and who the very poor teachers are. What’s more—and this is the finding that has galvanized the educational world—the difference between good teachers and poor teachers turns out to be vast.

Eric Hanushek, an economist at Stanford, estimates that the students of a very bad teacher will learn, on average, half a year’s worth of material in one school year. The students in the class of a very good teacher will learn a year and a half’s worth of material. That difference amounts to a year’s worth of learning in a single year. Teacher effects dwarf school effects: your child is actually better off in a “bad” school with an excellent teacher than in an excellent school with a bad teacher. Teacher effects are also much stronger than class-size effects. You’d have to cut the average class almost in half to get the same boost that you’d get if you switched from an average teacher to a teacher in the eighty-fifth percentile. And remember that a good teacher costs as much as an average one, whereas halving class size would require that you build twice as many classrooms and hire twice as many teachers.

Hanushek recently did a back-of-the-envelope calculation about what even a rudimentary focus on teacher quality could mean for the United States. If you rank the countries of the world in terms of the academic performance of their schoolchildren, the U.S. is just below average, half a standard deviation below a clump of relatively high-performing countries like Canada and Belgium. According to Hanushek, the U.S. could close that gap simply by replacing the bottom six per cent to ten per cent of public-school teachers with teachers of average quality. After years of worrying about issues like school funding levels, class size, and curriculum design, many reformers have come to the conclusion that nothing matters more than finding people with the potential to be great teachers. But there’s a hitch: no one knows what a person with the potential to be a great teacher looks like. The school system has a quarterback problem.

In my experience as a university professor I find that most colleagues think of themselves as above-average teachers, even when they are not. Essentially no "value-added" analysis is ever done, so people can have a 30 year teaching career without ever realizing that they aren't effective in the classroom. I've done many dozens of business presentations, to venture capitalists, technology partners, customers, analysts and even potential M&A acquirers, which has helped me improve my own teaching and communication skills. Despite the business setting such meetings are 90 percent teaching -- trying to convey key points to the audience in a limited time. I'm usually there with a team and my team isn't shy about telling me afterwards what worked and what didn't work, so I've had a lot of honest feedback that most professors never get.



The New Yorker cartoon and article capture some essential aspects of teaching and communication that are not widely understood. The teacher has to be simultaneously on top of the material itself and aware of what the class is doing / thinking / confused about. The big neglected factors in teaching are the ability to be a kind of air traffic controller (or symphony conductor) for the class, and the ability to empathize with (read the mind of) an individual student, to see what, exactly, is confusing them.

Are we getting smarter? Why?

It is well known that raw scores on IQ tests have been increasing at a rate of what would be about 3 IQ points per decade (the so-called Flynn effect). This means that, were the result of the test not rescaled so that the average is 100 by definition, the average IQ would have risen to 130 over the last century - i.e., the average person today scores better than all but 2 percent or so of the population in 1900. Malcolm Gladwell discusses this effect and its possible causes in this week's New Yorker, reviewing a new book claiming that modern society, with its fast-paced multimedia entertainment (including video games, computers, TV, etc.) actually improves our cognitive skills.

Gladwell: "Twenty years ago, a political philosopher named James Flynn uncovered a curious fact. Americans—at least, as measured by I.Q. tests—were getting smarter. This fact had been obscured for years, because the people who give I.Q. tests continually recalibrate the scoring system to keep the average at 100. But if you took out the recalibration, Flynn found, I.Q. scores showed a steady upward trajectory, rising by about three points per decade, which means that a person whose I.Q. placed him in the top ten per cent of the American population in 1920 would today fall in the bottom third. Some of that effect, no doubt, is a simple by-product of economic progress: in the surge of prosperity during the middle part of the last century, people in the West became better fed, better educated, and more familiar with things like I.Q. tests. But, even as that wave of change has subsided, test scores have continued to rise—not just in America but all over the developed world. What’s more, the increases have not been confined to children who go to enriched day-care centers and private schools. The middle part of the curve—the people who have supposedly been suffering from a deteriorating public-school system and a steady diet of lowest-common-denominator television and mindless pop music—has increased just as much. What on earth is happening? In the wonderfully entertaining “Everything Bad Is Good for You” (Riverhead; $23.95), Steven Johnson proposes that what is making us smarter is precisely what we thought was making us dumber: popular culture.

...As Johnson points out, television is very different now from what it was thirty years ago. It’s harder. A typical episode of “Starsky and Hutch,” in the nineteen-seventies, followed an essentially linear path: two characters, engaged in a single story line, moving toward a decisive conclusion. To watch an episode of “Dallas” today is to be stunned by its glacial pace—by the arduous attempts to establish social relationships, by the excruciating simplicity of the plotline, by how obvious it was. A single episode of “The Sopranos,” by contrast, might follow five narrative threads, involving a dozen characters who weave in and out of the plot. Modern television also requires the viewer to do a lot of what Johnson calls “filling in,” as in a “Seinfeld” episode that subtly parodies the Kennedy assassination conspiracists, or a typical “Simpsons” episode, which may contain numerous allusions to politics or cinema or pop culture. The extraordinary amount of money now being made in the television aftermarket—DVD sales and syndication—means that the creators of television shows now have an incentive to make programming that can sustain two or three or four viewings.

...It doesn’t seem right, of course, that watching “24” or playing a video game could be as important cognitively as reading a book. Isn’t the extraordinary success of the “Harry Potter” novels better news for the culture than the equivalent success of “Grand Theft Auto III”? Johnson’s response is to imagine what cultural critics might have said had video games been invented hundreds of years ago, and only recently had something called the book been marketed aggressively to children: (Johnson) Reading books chronically understimulates the senses. Unlike the longstanding tradition of gameplaying—which engages the child in a vivid, three-dimensional world filled with moving images and musical sound-scapes, navigated and controlled with complex muscular movements—books are simply a barren string of words on the page. . . .

Books are also tragically isolating. While games have for many years engaged the young in complex social relationships with their peers, building and exploring worlds together, books force the child to sequester him or herself in a quiet space, shut off from interaction with other children. . . .

But perhaps the most dangerous property of these books is the fact that they follow a fixed linear path. You can’t control their narratives in any fashion—you simply sit back and have the story dictated to you. . . . This risks instilling a general passivity in our children, making them feel as though they’re powerless to change their circumstances. Reading is not an active, participatory process; it’s a submissive one.

He’s joking, of course, but only in part. The point is that books and video games represent two very different kinds of learning. When you read a biology textbook, the content of what you read is what matters. Reading is a form of explicit learning. When you play a video game, the value is in how it makes you think. Video games are an example of collateral learning, which is no less important."

Gut instinct, or efficient markets?

Slate is running an interesting dialog between Malcolm Gladwell and James Surowiekci, two of the New Yorker's best writers, and authors of the recent books Blink and The Wisdom of Crowds.

Blink is about instances when a snap judgement (relying on murky subconscious processes) may actually be superior to reasoned decision making. Surowiecki's book documents cases where the aggregate wisdom of many individuals is superior to that of a lone expert (the efficient market!). If the two notions seem hard to reconcile, join the club and read the dialog.

Surowiecki's thesis is one I've thought a lot about. In instances where the answer to a problem takes a fairly simple form ("What is the value of a share of Microsoft?" or "How many pennies are in this jar?"), it is plausible that the aggregation of all opinions eliminates biases in a useful way. While the market may be far from completely efficent, it is nevertheless hard to beat it consistently. (Don't try this with questions like "How do we reconcile quantum mechanics with general relativity?", though. In that case it might be better to ask the acknowledged expert than the crowd.)

On the other hand, I suspect that in many cases people's intuition is just plain bad, and systematic reasoning offers better results. (At least, that's my gut instinct ;-) For example, it is well documented from behavioral finance that people have a very bad innate understanding of probabilities and risk. This doesn't exclude the possibility that for certain questions, like "can I trust this guy with my girlfriend?", perhaps evolution has provided us with a lot of hard-wired machinery to quickly deduce the answer.

From David Brooks' review of Blink in the Times:
"Gladwell says we are thin-slicing all the time -- when we go on a date, meet a prospective employee, judge any situation. We take a small portion of a person or problem and extrapolate amazingly well about the whole. A psychologist named Nalini Ambady gave students three 10-second soundless videotapes of a teacher lecturing. Then she asked the students to rate the teacher. Their ratings matched the ratings from students who had taken the teacher's course for an entire semester. Then she cut the videotape back to two seconds and showed it to a new group. The ratings still matched those of the students who'd sat through the entire term."

Note added: the last election combined elements of Blinkish thin-slicing, as well as purported Wisdom of Crowds. It looks like the crowd trusted their gut instincts (rather than systematic reasoning?) and elected Bush. We'll see whether that was a better decision than an "expert" like me would have made ;-)

Gladwell amongst the patent trolls

Malcolm Gladwell writes about Nathan Myhrvold's company Intellectual Ventures in the recent New Yorker. (Myhrvold is the former cosmologist who left physics and eventually became consigliere to Bill Gates, founding Microsoft Research and charting Microsoft's blue sky research direction. He famously missed the importance of the Internet until the mid 90's.) If you read this blog often, you know my opinion about Gladwell: he has a good nose for interesting topics, but not enough brainpower or common sense for reliable analysis. The same is true here: he produces an interesting profile of Myhrvold (although see here for a much better one from 1997 by Ken Auletta) and friends, but seems to entirely miss a number of important points. Intellectual Ventures is not about real inventions, but about patenting around ideas so that they have a future claim on the ones that turn out the be useful. In other words, they are patent trolls. Gladwell does not seem to realize the difference between rampant speculation and true invention: the hours of painstaking work in the lab required to convert an idea into reality.

Here's an excerpt about how the "invention" process works -- get some smart guys in a room and let them talk (every theory group lounge is a fount of commercializable ideas ;-). Yes! if your inventors are smart enough, they can produce 36 new inventions at dinner! Is this a statement about real innovation, or about what a patent attorney might manage to get the understaffed, overburdened USPTO to approve? It makes a mockery of what real inventors and innovators do. Why start a company and hire engineers to build a prototype? Just get a few lawyers and patent everything in sight...

How useful is it to have a group of really smart people brainstorm for a day? When Myhrvold started out, his expectations were modest. Although he wanted insights like Alexander Graham Bell’s, Bell was clearly one in a million, a genius who went on to have ideas in an extraordinary number of areas—sound recording, flight, lasers, tetrahedral construction, and hydrofoil boats, to name a few. ...

But then, in August of 2003, I.V. held its first invention session, and it was a revelation. “Afterward, Nathan kept saying, ‘There are so many inventions,’ ” Wood recalled. “He thought if we came up with a half-dozen good ideas it would be great, and we came up with somewhere between fifty and a hundred. I said to him, ‘But you had eight people in that room who are seasoned inventors. Weren’t you expecting a multiplier effect?’ And he said, ‘Yeah, but it was more than multiplicity.’ Not even Nathan had any idea of what it was going to be like.”

The original expectation was that I.V. would file a hundred patents a year. Currently, it’s filing five hundred a year. It has a backlog of three thousand ideas. Wood said that he once attended a two-day invention session presided over by Jung, and after the first day the group went out to dinner. “So Edward took his people out, plus me,” Wood said. “And the eight of us sat down at a table and the attorney said, ‘Do you mind if I record the evening?’ And we all said no, of course not. We sat there. It was a long dinner. I thought we were lightly chewing the rag. But the next day the attorney comes up with eight single-spaced pages flagging thirty-six different inventions from dinner. Dinner.”

For the cognoscenti out there, yes, the Wood mentioned in the article is none other than Star Warrior Lowell Wood, former head of the zany (useless?) O Group (NYTimes 1984) at Livermore. Wood is perfect for Myhrvold's purposes -- for decades his group bamboozled the US defense establishment with wild ideas that cost taxpayers billions of dollars. Follow the link to the Times article and tell me how many of the ideas mentioned turned into something useful, now almost a quarter century later.

Rather than leave you with a completely negative impression of the article, I include the following excerpt, which has Wood noticing something about cancer cells in the bloodstream that seems to have eluded biologists and medical researchers for some time. It is true that there are great ideas out there just waiting to be discovered, but lots of people can have the same idea. The hard part is making the idea into a practical, commercially viable reality.

...Last March, Myhrvold decided to do an invention session with Eric Leuthardt and several other physicians in St. Louis. Rod Hyde came, along with a scientist from M.I.T. named Ed Boyden. Wood was there as well.

“Lowell came in looking like the Cheshire Cat,” Myhrvold recalled. “He said, ‘I have a question for everyone. You have a tumor, and the tumor becomes metastatic, and it sheds metastatic cancer cells. How long do those circulate in the bloodstream before they land?’ And we all said, ‘We don’t know. Ten times?’ ‘No,’ he said. ‘As many as a million times.’ Isn’t that amazing? If you had no time, you’d be screwed. But it turns out that these cells are in your blood for as long as a year before they land somewhere. What that says is that you’ve got a chance to intercept them.”

How did Wood come to this conclusion? He had run across a stray fact in a recent issue of The New England Journal of Medicine. “It was an article that talked about, at one point, the number of cancer cells per millilitre of blood,” he said. “And I looked at that figure and said, ‘Something’s wrong here. That can’t possibly be true.’ The number was incredibly high. Too high. It has to be one cell in a hundred litres, not what they were saying—one cell in a millilitre. Yet they spoke of it so confidently. I clicked through to the references. It was a commonplace. There really were that many cancer cells.”

Wood did some arithmetic. He knew that human beings have only about five litres of blood. He knew that the heart pumps close to a hundred millilitres of blood per beat, which means that all of our blood circulates through our bloodstream in a matter of minutes. The New England Journal article was about metastatic breast cancer, and it seemed to Wood that when women die of metastatic breast cancer they don’t die with thousands of tumors. The vast majority of circulating cancer cells don’t do anything.

“It turns out that some small per cent of tumor cells are actually the deadly ones,” he went on. “Tumor stem cells are what really initiate metastases. And isn’t it astonishing that they have to turn over at least ten thousand times before they can find a happy home? You naïvely think it’s once or twice or three times. Maybe five times at most. It isn’t. In other words, metastatic cancer—the brand of cancer that kills us—is an amazingly hard thing to initiate. Which strongly suggests that if you tip things just a little bit you essentially turn off the process.”

That was the idea that Wood presented to the room in St. Louis. From there, the discussion raced ahead. Myhrvold and his inventors had already done a lot of thinking about using tiny optical filters capable of identifying and zapping microscopic particles. They also knew that finding cancer cells in blood is not hard. They’re often the wrong size or the wrong shape. So what if you slid a tiny filter into a blood vessel of a cancer patient? “You don’t have to intercept very much of the blood for it to work,” Wood went on. “Maybe one ten-thousandth of it. The filter could be put in a little tiny vein in the back of the hand, because that’s all you need. Or maybe I intercept all of the blood, but then it doesn’t have to be a particularly efficient filter.”

Wood was a physicist, not a doctor, but that wasn’t necessarily a liability, at this stage. “People in biology and medicine don’t do arithmetic,” he said. He wasn’t being critical of biologists and physicians: this was, after all, a man who read medical journals for fun. He meant that the traditions of medicine encouraged qualitative observation and interpretation. But what physicists do—out of sheer force of habit and training—is measure things and compare measurements, and do the math to put measurements in context. At that moment, while reading The New England Journal, Wood had the advantages of someone looking at a familiar fact with a fresh perspective.

That was also why Myhrvold had wanted to take his crew to St. Louis to meet with the surgeons. He likes to say that the only time a physicist and a brain surgeon meet is when the physicist is about to be cut open—and to his mind that made no sense. Surgeons had all kinds of problems that they didn’t realize had solutions, and physicists had all kinds of solutions to things that they didn’t realize were problems. At one point, Myhrvold asked the surgeons what, in a perfect world, would make their lives easier, and they said that they wanted an X-ray that went only skin deep. They wanted to know, before they made their first incision, what was just below the surface. When the Intellectual Ventures crew heard that, their response was amazement. “That’s your dream? A subcutaneous X-ray? We can do that.”

Let me close with my usual observation (specifically aimed at venture capitalists, research lab directors and university administrators) concerning an asymmetry in cognitive depth: yes, physicists can casually read the New England Journal of Medicine and come up with interesting insights, but, no, biologists and medical doctors cannot read Physical Review.

Earnings effects of personality, education and IQ for the gifted

Thanks to a reader for pointing me to this recent paper by Heckman and collaborators, which makes use of data from the Terman study of gifted individuals (minimum IQ of 135 on the Stanford-Binet).

Of the personality factors, Conscientiousness and Extraversion had the largest (positive) effect on lifetime earnings: the most conscientious or extraverted individuals earned, on average, about 50% more than the least (see figures below). See here for more on Big 5 personality factors and a link to a personality test.

The Effects of Education, Personality, and IQ on Earnings of High-Ability Men

This paper estimates the internal rate of return (IRR) to education for men and women of the Terman sample, a 70-year long prospective cohort study of high-ability individuals. The Terman data is unique in that it not only provides full working-life earnings histories of the participants, but it also includes detailed profiles of each subject, including IQ and measures of latent personality traits. Having information on latent personality traits is significant as it allows us to measure the importance of personality on educational attainment and lifetime earnings.

Our analysis addresses two problems of the literature on returns to education: First, we establish causality of the treatment effect of education on earnings by implementing generalized matching on a full set of observable individual characteristics and unobserved personality traits. Second, since we observe lifetime earnings data, our estimates of the IRR are direct and do not depend on the assumptions that are usually made in order to justify the interpretation of regression coefficients as rates of return.

For the males, the returns to education beyond high school are sizeable. For example, the IRR for obtaining a bachelor's degree over a high school diploma is 11.1%, and for a doctoral degree over a bachelor's degree it is 6.7%. These results are unique because they highlight the returns to high-ability and high-education individuals, who are not well-represented in regular data sets.

Our results highlight the importance of personality and intelligence on our outcome variables. We find that personality traits similar to the Big Five personality traits are significant factors that help determine educational attainment and lifetime earnings. Even holding the level of education constant, measures of personality traits have significant effects on earnings. Similarly, IQ is rewarded in the labor market, independently of education. Most of the effect of personality and IQ on life-time earnings arise late in life, during the prime working years. Therefore, estimates from samples with shorter durations underestimate the treatment effects.

Here are a couple of interesting excerpts from the paper:

... Our third contribution is to show how the effect of personality on earnings varies through-out the men’s working lives. We find that without access to long follow-up data, the estimated effect would be understated. Note that even though the Terman sample has a restricted range of IQ, there is substantial variation in personality. In fact, the Terman men do not differ from the general population in terms of personality.

... note that even when controlling for rich background variables, IQ maintains a statistically significant effect on lifetime earnings. Even though the effect is slightly diminished from the un-controlled association of the first column, it is still sizable. Malcolm Gladwell claims rather generally in his book Outliers that for the Terman men, IQ did not matter once family background and other observable personal characteristics were taken into account. While we do not want to argue that IQ has a larger role for the difference between 50 and 100, for example, than for the difference between 150 and 200, we do want to point out that even at the high end of the ability distribution, IQ has meaningful consequences. [The syntax of this last sentence is strange. Presumably the impact of IQ variation from 50 to 100 (from severely handicapped to average) is larger than for 150 to 200, even though their results show a significant effect even in the very high range.]

Below are some nice figures (click for larger versions). Note the personality factor distribution among Termites was similar to that of the overall population, whereas the IQ range was restricted due to selection. Typical lifetime earnings for this group of exceptionally able men ranged from $2 to $3 million in 2008 dollars.




Compare the bottom right IQ graph with SMPY results which show the impact of ability (SAT-M measured before age 13) on publication and patent rates. Ability in the SMPY graph varies between 99th and 99.99th percentile in quartiles Q1-Q4. The variation in IQ between the bottom and top deciles of the Terman study covers a similar range. The Terman super-smarties (i.e., +4 SD) only earned slightly more (say, 15-20% over a lifetime) than the ordinary smarties (i.e., +2.5 SD), but the probability of earning a patent (SMPY) went up by about 4x over the corresponding ability range.

Life impacts of personality and intelligence

Here are two nice figures I came across recently.

The first, based on SMPY data, displays odds ratios for various accomplishments (doctorate degree, STEM publication, patent, high income, tenure) as a function of SAT-M score at age 13. The quartiles correspond roughly to 1 in 200 ability (Q1) to 1 in 10k ability (Q4). This data soundly refutes the "IQ above 120 doesn't matter" Malcolm Gladwell nonsense. See earlier post Horsepower matters; Psychometrics works.

In (imprecise) words: "Profoundly gifted children are 10+ times more likely than merely gifted children to, e.g., earn a patent or gain tenure at a top research university. They are at least several times more likely to earn exceptionally high incomes." (Note "merely gifted" is somewhat below the Q1 SMPY cut -- most school systems use top few percent vs top 0.5 percent.)

The second figure shows regression coefficients of income (at various ages) vs IQ and personality traits (standardized, so returns for each SD of trait). This was originally discussed in Earnings effects of personality, education and IQ for the gifted; see also this paper (Miriam Gensowski, Copenhagen). Note the IQ returns may be underestimated for average individuals since the data source is Terman and there is significant restriction of range (everyone tested at better than 1 in 200 or so on the Stanford-Binet). Nevertheless there are still positive returns to above average IQ within the Terman group (analogous to SMPY results above).

It pays to be Smart, Disciplined/Focused, Extraverted, and Mean! 8-(

Friday Night Lights

Over the break I had some time to catch up on the television series Friday Night Lights (FNL), thanks to Hulu.com. The show is loosely based on the book by H.G. Bissinger:

Friday Night Lights: A Town, a Team, and a Dream is a 1990 non-fiction book written by H. G. Bissinger. The book follows the story of the 1988 Permian High School Panthers football team from Odessa as they made a run towards the Texas state championship. While originally intended to be a Hoosiers-type chronicle of high school sports holding a small town together, the final book ended up being critical about life in the town of Odessa, Texas, complete with portraits of what Bissinger called "the ugliest racism" he has ever witnessed, as well as misplaced priorities, where football conquered most aspects of the town and academics were ignored for the sake of championships.

Bissinger was a sports writer for The Philadelphia Inquirer, when he decided to write a book about high school sports. After a search, he settled on Odessa, TX and their famous Permian Panthers. The Panthers had a long, rich history of winning in Texas' AAAA and AAAAA division, winning championships in 1965, 1972, 1980 and 1984 at the time when Bissinger and his family moved from Philadelphia to Odessa. He spent the entire football season with the Permian Panther players, their families, the coaches, and even many of the townspeople in an effort to understand the town and their football culture and what created such madness for their football team.

In 2002, Sports Illustrated named Friday Night Lights the fourth-greatest book ever written about sports.

Bissinger's book should be of interest to anyone who wants to understand small town American life and its microcosm, the local high school. If you like FNL, I also recommend The Courting of Marcus Dupree, by Willie Morris, about the recruiting of a superstar running back from Missisippi. (Link goes to Google Books version.) FNL didn't feature any real football talents -- although the team was very successful none of the players went on to big time college careers. Dupree on the other hand was one of the top high school backs of all time, breaking Herschel Walker's touchdown record. Read pages 34-44, which will teach you much more about outliers than anything written by Malcolm Gladwell. (Dupree, despite his small school background, had something that the Permian players, with their expensive facilities, highly paid coaching staff and unrivaled football mania, just couldn't match -- raw, god-given talent ;-)

NYTimes review: IN 1964, the town of Philadelphia, Miss., became the symbol of much that was wrong with America - if only by virtue of its having provided the setting for the murders of three young civil-rights workers, Michael Schwerner, Andrew Goodman and James Earl Chaney. Just 17 years later, in the autumn of 1981, Philadelphia became the focus of more benign attention. It was the site of a competition among the nation's leading college football powers to recruit the most highly touted high-school player in the country. ...

The FNL television series is fantastic, and has a devoted following despite mediocre ratings. To get a sense of it, have a look at the following clips. (Note, I am finding on Safari that the hulu embedded players don't work well, which is why I also link directly to the hulu pages where you can view the clips.)

The hard nosed side of big time high school sports. view



Family life in America today. view



Hardscrabble in Texas. view



If you liked those clips, watch this this full episode and this one.

Perverse Incentives and Replication in Science

Here's a depressing but all too common pattern in scientific research:

1. Study reports results which reinforce the dominant, politically correct, narrative.

2. Study is widely cited in other academic work, lionized in the popular press, and used to advance real world agendas.

3. Study fails to replicate, but no one (except a few careful and independent thinkers) notices.

For numerous examples, see, e.g., any of Malcolm Gladwell's books :-(

A recent example: the idea that collective intelligence of groups (i.e., ability to solve problems and accomplish assigned tasks) is not primarily dependent on the cognitive ability of individuals in the group.

It seems plausible to me that by adopting certain best practices for collaboration one can improve group performance, and that diversity of knowledge base and personal experience could also enhance performance on certain tasks. But recent results in this direction were probably oversold, and seem to have failed to replicate.

James Thompson has given a good summary of the situation.

Parts 1 and 2 of our story:

MIT Center for Collective Intelligence: ... group-IQ, or “collective intelligence” is not strongly correlated with the average or maximum individual intelligence of group members but is correlated with the average social sensitivity of group members, the equality in distribution of conversational turn-taking, and the proportion of females in the group.

Is it true? The original paper on this topic, from 2010, has been cited 700+ times. See here for some coverage on this blog when it originally appeared.

Below is the (only independent?) attempt at replication, with strongly negative results. The first author is a regular (and very insightful) commenter here -- I hope he'll add his perspective to the discussion. Have we reached part 3 of the story?

Smart groups of smart people: Evidence for IQ as the origin of collective intelligence in the performance of human groups

Timothy C. Bates a,b,⁎, Shivani Gupta a
a Department of Psychology, University of Edinburgh
b Centre for Cognitive Ageing and Cognitive Epidemiology, University of Edinburgh

What allows groups to behave intelligently? One suggestion is that groups exhibit a collective intelligence accounted for by number of women in the group, turn-taking and emotional empathizing, with group-IQ being only weakly-linked to individual IQ (Woolley, Chabris, Pentland, Hashmi, & Malone, 2010). Here we report tests of this model across three studies with 312 people. Contrary to prediction, individual IQ accounted for around 80% of group-IQ differences. Hypotheses that group-IQ increases with number of women in the group and with turn-taking were not supported. Reading the mind in the eyes (RME) performance was associated with individual IQ, and, in one study, with group-IQ factor scores. However, a well-fitting structural model combining data from studies 2 and 3 indicated that RME exerted no influence on the group-IQ latent factor (instead having a modest impact on a single group test). The experiments instead showed that higher individual IQ enhances group performance such that individual IQ determined 100% of latent group-IQ. Implications for future work on group-based achievement are examined.

From the paper:

Given the ubiquitous importance of group activities (Simon, 1997) these results have wide implications. Rather than hiring individuals with high cognitive skill who command higher salaries (Ritchie & Bates, 2013), organizations might select-for or teach social sensitivity thus raising collective intelligence, or even operate a female gender bias with the expectation of substantial performance gains. While the study has over 700 citations and was widely reported to the public (Woolley, Malone, & Chabris, 2015), to our knowledge only one replication has been reported (Engel, Woolley, Jing, Chabris, & Malone, 2014). This study used online (rather than in-person) tasks and did not include individual IQ. We therefore conducted three replication studies, reported below.

... Rather than a small link of individual IQ to group-IQ, we found that the overlap of these two traits was indistinguishable from 100%. Smart groups are (simply) groups of smart people. ... Across the three studies we saw no significant support for the hypothesized effects of women raising (or men lowering) group-IQ: All male, all female and mixed-sex groups performed equally well. Nor did we see any relationship of some members speaking more than others on either higher or lower group-IQ. These findings were weak in the initial reports, failing to survive incorporation of covariates. We attribute these to false positives. ... The present findings cast important doubt on any policy-style conclusions regarding gender composition changes cast as raising cognitive-efficiency. ...

In conclusion, across three studies groups exhibited a robust cognitive g-factor across diverse tasks. As in individuals, this g-factor accounted for approximately 50% of variance in cognition (Spearman, 1904). In structural tests, this group-IQ factor was indistinguishable from average individual IQ, and social sensitivity exerted no effects via latent group-IQ. Considering the present findings, work directed at developing group-IQ tests to predict team effectiveness would be redundant given the extremely high utility, reliability, validity for this task shown by individual IQ tests. Work seeking to raise group-IQ, like re- search to raise individual IQ might find this task achievable at a task- specific level (Ritchie et al., 2013; Ritchie, Bates, & Plomin, 2015), but less amenable to general change than some have anticipated. Our attempt to manipulate scores suggested that such interventions may even decrease group performance. Instead, work understanding the developmental conditions which maximize expression of individual IQ (Bates et al., 2013) as well as on personality and cultural traits supporting cooperation and cumulation in groups should remain a priority if we are to understand and develop cognitive ability. The present experiments thus provide new evidence for a central, positive role of individual IQ in enhanced group-IQ.

Meta-Observation: Given the 1-2-3 pattern described above, one should be highly skeptical of results in many areas of social science and even biomedical science (see link below). Serious researchers (i.e., those who actually aspire to participate in Science) in fields with low replication rates should (as a demonstration of collective intelligence!) do everything possible to improve the situation. Replication should be considered an important research activity, and should be taken seriously.

Most researchers I know in the relevant areas have not yet grasped that there is a serious problem. They might admit that "some studies fail to replicate" but don't realize the fraction might be in the 50 percent range!

More on the replication crisis in certain fields of science.

New Yorker: Practice Doesn't Make Perfect (Zach Hambrick, MSU Psychology)

MSU Psychology Professor Zach Hambrick is featured in this New Yorker article about Nature vs Nurture. How far the pendulum has swung since the naive days of Malcolm Gladwell's Outliers and its credulous embrace of Anders Ericsson's nurturist claims. David Lubinski and SMPY also make an appearance.

New Yorker: Practice Doesn't Make Perfect

... So how much did practice actually explain? In a 2014 meta-analysis that looked specifically at the relationship between deliberate practice and performance in music, games like chess, sports, education, and other professions, Hambrick and his team found a relationship that was even more complex than they had expected. For some things, like games, practice explained about a quarter of variance in expertise. For music and sports, the explanatory power accounted for about a fifth. But for education and professions like computer science, military-aircraft piloting, and sales, the effect ranged from small to tiny. For all of these professions, you obviously need to practice, but natural abilities matter more.

What’s more, the explanatory power of practice fell even further when Hambrick took exact level of expertise into account. In sports—one of the areas in which deliberate practice seems to make the most difference—it turned out that the more advanced the athlete, the less of a role practice plays. Training an average athlete for a set number of hours yields far more results than training an élite athlete, which, in turn, yields greater results than training a super-élite athlete. Put differently, someone like me is going to improve a great deal with even a few hundred hours of training. But within an Olympic team tiny differences in performance are unlikely to be the result of training: these athletes train together, with the same coach, day in and day out. Those milliseconds come from somewhere else. Some may be due to the fact that genetic differences can account for some of the response to training. ...

So where else, exactly, do performance differences come from? While Hambrick’s work has been focussed more explicitly on practice and genetics, David Lubinski, a professor of psychology at Vanderbilt University, has been approaching the question from a slightly different angle: through what’s called the Study of Mathematically Precocious Youth (smpy), a longitudinal study of the lives of students who, by the age of thirteen, had scored in the top one per cent of mathematical-reasoning ability and were then selected to take part in an enriched educational environment. (The study, co-directed for many years by Lubinski and his wife, Vanderbilt’s education-school dean, Camilla Benbow, was described in detail in a recent article in Nature.) It’s a crucial supplement to work like Hambrick’s; the data you get from close observation of the same sample and the same individuals over time can answer questions other approaches can’t. “What kinds of practice are more effective? What approaches more effective for some people than others?” Hambrick asks. “We need all the pieces to the puzzle to maximize people’s potential. Lubinski’s work on mathematically precocious youth is an essential piece.”

Annals of psychometry: IQs of eminent scientists

[ See this 2016 post for the original papers / book and the identities of the 64 scientists. ]

I recently came across a 1950s study of eminent scientists by Harvard psychologist Anne Roe: The Making of a Scientist (1952). Her study is by far the most systematic and sophisticated that I am aware of. She selected 64 eminent scientists -- well known, but not quite at the Nobel level -- in a more or less random fashion, using, e.g., membership lists of scholarly organizations and expert evaluators in the particular subfields. Roughly speaking, there were three groups: physicists (divided into experimental and theoretical subgroups), biologists (including biochemists and geneticists) and social scientists (psychologists, anthropologists).

The Making of a Scientist devotes only one chapter to psychometrics. The other chapters describe the motivation for the study, how the 64 scientists were selected, interviews with the scientists, details of their family history, work life, etc.

Roe devised her own high-end intelligence tests as follows: she obtained difficult problems in verbal, spatial and mathematical reasoning from the Educational Testing Service, which administers the SAT, but also performs bespoke testing research for, e.g., the US military. Using these problems, she created three tests (V, S and M), which were administered to the 64 scientists, and also to a cohort of PhD students at Columbia Teacher's College. The PhD students also took standard IQ tests and the results were used to norm the high-end VSM tests using an SD = 15. Most IQ tests are not good indicators of true high level ability (e.g., beyond +3 SD or so).

Average ages of subjects: mid-40s for physicists, somewhat older for other scientists

Overall normed scores:

Test (Low / Median / High)

V 121 / 166 / 177

S 123 / 137 / 164

M 128 / 154 / 194

Roe comments: (1) V test was too easy for some takers, so top score no ceiling. (2) S scores tend to decrease with age (correlation .4). Peak (younger) performance would have been higher. (3) M test was found to be too easy for the physicists; only administered to other groups.

It is unlikely that any single individual obtained all of the low scores, so each of the 64 would have been strongly superior in at least one or more areas.

Median scores (raw) by group:

group (V / S / M)

Biologists 56.6 / 9.4 / 16.8
Exp. Physics 46.6 / 11.7 / *
Theo. Physics 64.2 / 13.8 / *
Psychologists 57.7 / 11.3 / 15.6
Anthropologists 61.1 / 8.2 / 9.2

The lowest score in each category among the 12 theoretical physicists would have been roughly V 160 (!) S 130 M >> 150. (Ranges for all groups are given, but I'm too lazy to reproduce them here.) It is hard to estimate the M scores of the physicists since when Roe tried the test on a few of them they more or less solved every problem modulo some careless mistakes. Note the top raw scores (27 out of 30 problems solved) among the non-physicists (obtained by 2 geneticists and a psychologist), are quite high but short of a full score. The corresponding normed score is 194!

The lowest V scores in the 120-range were only obtained by 2 experimental physicists, all other scientists scored well above this level -- note the median is 166.

My comments:

The results strongly suggest that high IQ provides a significant advantage in science. It is sometimes claimed that IQ is irrelevant beyond a threshold: more precisely, that the advantage conferred by IQ above some threshold (e.g., 120) decreases significantly as other factors like drive or creativity take precedence. But, if that were the case it would be unlikely to have found such high scores in this group. The average IQ of a science PhD is roughly 130, and individuals with IQs in the higher range described above constitute a tiny fraction of all scientists. If IQ were irrelevant above 130 we would expect the most eminent group to have an average similar to the overall population of scientists.

Conversely, I think one should be impressed that a simple test which can be administered in a short period of time (e.g., 30 minutes for Roe's high-end exams) offers significant predictive power. While it is not true that anyone with a high IQ can or will become a great scientist (certainly other factors like drive, luck, creativity play a role), one can nevertheless easily identify the 95 percent (even 99 percent) of the population for whom success in science is highly improbable. Psychometrics works!

The scores for theoretical physicists confirm an estimate made to me by a famous colleague many years ago, that only 1 in 100,000 people could do high level theoretical physics.

Feynman's 124: in this context one often hears of Feynman's modest grade school IQ score of 124. To understand this score we have to remember that typical IQ tests (e.g., administered to public school children) tend to have low ceilings. They are not of the kind that Roe used in her study. One can imagine that the ceiling on Feynman's exam was roughly 135 (say, 99th percentile). If Feynman received the highest score on the mathematical portion, and a modest score of 115 on the verbal, we can easily understand the resulting average of 124. However, it is well known that Feynman was extremely strong mathematically. He was asked on short notice to take the Putnam exam for MIT as a senior, and received the top score in the country that year! On Roe's test Feynman's math score would presumably have been > 190, with a correspondingly higher composite IQ.

I thought I should put this post up now, as the new book by Malcolm Gladwell, Outliers: Why Some People Succeed and Some Don’t is out soon and will surely handicap the discourse on this subject for years to come :-)