More on psychometrics

I've had some email discussions elaborating on the psychometrics slides I posted earlier. The slides themselves don't convey a lot of the important points I made in the talks so I thought I'd share this message on the blog.

Hi Guys,

I'm very interested in exactly the question Henry is getting at.

I think our simple two factor model

Grades = ability + work ethic = IQ + W

is not too crazy. Note that once you fix the ability level (=SAT score) the remaining variance in GPA has about the same SD regardless of value of SAT score (vertical red lines in the big figure in the slides). That suggests that we can think of IQ and W as largely uncorrelated random variables -- so there are smart lazy people, hard working dumb people, etc. I can't really prove the residual variance after IQ is controlled for is due to work ethic, but my experience in the classroom suggests that it is. (Note work ethic here isn't necessary general work ethic as a personality factor, but how hard the kid worked in the specific course. However, in our data we average over many courses taken by many kids, so perhaps it does get at variation of personality factor(s) in the overall population.) Beyond work ethic, some people are just more "effective" -- they can get themselves organized, are disciplined, can adapt to new challenges, are emotionally robust -- and this is also absorbed in the W factor above.

Now, in some fields there seems to be a minimum cognitive threshold. I've known physics students who worked incredibly hard and just couldn't master the material. That is reflected in our data on pure math and physics majors at UO. For all majors there is a significant positive correlation between SAT and upper GPA (in the range .3-.5).

Whether IQ has a large impact on life outcomes depends on how you ask the question. I do believe that certain professions are almost off-limits for people below a certain IQ threshold. But for most jobs (even engineer or doctor), this threshold is surprisingly low IF the person has a strong work ethic. In other words a +1 SD IQ person can probably still be a doctor or engineer if they have +(2-3) SD work ethic. However, such people, if they are honest with themselves, understand that they have some cognitive disadvantages relative to their peers. I've chosen a profession in which, every so often, I am the dumbest guy in the room -- in fact I put myself in this situation by going to workshops and wanting to talk to the smartest guys I can find :-) For someone of *average* work ethic I think you can easily find jobs for which the IQ threshold is +2 SD or higher. The typical kid admitted to grad school in my middle-tier physics department is probably > +2 SD IQ and at least +1.5 SD in work ethic -- ditto for a top tier law or med school. That's probably also the case these days for any "academic admit" at a top Ivy.

For typical jobs I think the correlation between success/income and IQ isn't very high. Other factors come into play, like work ethic, interpersonal skills, affect, charisma, luck, etc. This may even be true in many "elite" professions once you are talking about a population where everyone is above the minimum IQ threshold -- if returns to IQ above threshold are not that large then the other factors dominate and determine level of success. What is interesting about the Roe and SMPY studies is that they suggest that in science the returns to IQ above the +2 SD threshold (for getting a PhD) are pretty high. ***

Henry is right that for ideological reasons many researchers are happy to present the data so as to minimize the utility of IQ or testing in making life predictions. They might even go so far as to claim that since we use g-loaded tests in admissions, the conclusion that some professions require high IQ is actually circular. The social scientist who walked out of my Sci Foo talk actually made that claim.

Finally, when it comes to *individual* success I think most analysts significantly underestimate the role of pure blind luck (i.e., what remains when all other reasonable, roughly measurable variables have been accounted for; of course this averages out of any large population study). Or perhaps I am just reassuring myself about my limited success in life :-)


Steve

PS In the actual talks I gave I made most of these points. The slides are kind of bare bones...


*** You would have difficulty finding a hard scientist who would disagree with the statement it is a big advantage in my field to be super smart. However, thanks to political correctness, social science indoctrination, or unfamiliarity with psychometrics, it IS common for scientists to deny that being super smart has anything to do with scores on IQ tests. I myself question the validity of IQ tests beyond +(3-4) SD -- I'm more impressed by success on the IMO, Putnam, or in other high level competitions. (Although I realize that training has a big impact on performance in these competitions I do think real talent is a necessary condition for success.)