Wednesday, September 14, 2011

True grit

The next project with my colleague Jim Schombert is to see whether student personality inventories increase our ability to predict college GPA. In previous work we found a .4 or so correlation between SAT score and upper division in-major GPA. (Other studies, which focus on freshman GPA, typically find lower correlations but this is partly because academically stronger freshmen usually take harder courses. At the upper division level, majors typically have to take certain core courses, so there is more uniformity.) The correlation is somewhat higher (.5 to .6) if we use a z score derived from high school GPA and SAT. We think GPA is a proxy for conscientiousness, or what is referred to below as grit. But there is too much grade inflation in high school these days, and GPA depends both on work ethic and cognitive ability. So we'd like to see how well personality variables work. Optimistically, I think we can do better than correlations of .6, which is pretty impressive for social science.

NYTimes: ... People who accomplished great things, she noticed, often combined a passion for a single mission with an unswerving dedication to achieve that mission, whatever the obstacles and however long it might take. She decided she needed to name this quality, and she chose the word “grit.”

She developed a test to measure grit, which she called the Grit Scale. It is a deceptively simple test, in that it requires you to rate yourself on just 12 questions, from “I finish whatever I begin” to “I often set a goal but later choose to pursue a different one.” It takes about three minutes to complete, and it relies entirely on self-report — and yet when Duckworth took it out into the field, she found it was remarkably predictive of success. At Penn, high grit ratings allowed students with relatively low college-board scores to nonetheless achieve high G.P.A.’s. Duckworth and her collaborators gave their grit test to more than 1,200 freshman cadets as they entered West Point and embarked on the grueling summer training course known as Beast Barracks. The military has developed its own complex evaluation, called the Whole Candidate Score, to judge incoming cadets and predict which of them will survive the demands of West Point; it includes academic grades, a gauge of physical fitness and a Leadership Potential Score. But at the end of Beast Barracks, the more accurate predictor of which cadets persisted and which ones dropped out turned out to be Duckworth’s 12-item grit questionnaire.

[Of course, the grit score can't be used for admissions -- too easy to game, unlike an IQ test! Instead we use proxies for grit, like extracurriculars.]

... The first question Duckworth addressed, again, was the relative importance of I.Q. and self-control. She and her team of researchers gave middle-school students at Riverdale and KIPP a variety of psychological and I.Q. tests. They found that at both schools, I.Q. was the better predictor of scores on statewide achievement tests, but measures of self-control were more reliable indicators of report-card grades.

Duckworth’s research convinced Levin and Randolph that they should try to foster self-control and grit in their students. Yet those didn’t seem like the only character strengths that mattered. ... After a few small adjustments (Levin and Randolph opted to drop love in favor of curiosity), they settled on a final list: zest, grit, self-control, social intelligence, gratitude, optimism and curiosity.

Here's a figure from our paper Data Mining the University, which shows how SAT predicts GPA. The stars are over- and under-achievers (blue = male, red = female). Do the people in the upper left (over-achievers) have grit?

44 comments:

  1. Males and females should be analyzed separately if the intent is to make predictions beyond school GPA...

    ReplyDelete
  2. In our sample I would bet women average higher conscientiousness overall. This might be just an age thing -- men mature later.

    ReplyDelete
  3. Wengchung12:57 PM

    >This might be just an age thing -- men mature later.

    Are you certain this is true for emotional development post teenage years? Since you used the SAT, I'm going to assume the average age of your sample is around 17.

    As both males and females have both obviously reached cognitive potential at age 17 (both the typical age of an SAT taker and around the age Rushton believes the gender gaps in g become more evident).

    I had thought variations in certain personality traits between genders in early age (such as conscientiousness) would be influenced by both the IQ and cognitive profiles of the genders - meaning - that when the IQ of males raises about 3 points above females in the very late teenage years, the conscientiousness of males should spike in a similar manner

    A female lead in this trait in earlier age may dissipate to being equal, or even at a deficit in early adulthood.

    ReplyDelete
  4. ben_g2:35 PM

    "As both males and females have both obviously reached cognitive potential at age 17 (both the typical age of an SAT taker and around the age Rushton believes the gender gaps in g become more evident)."

    Heritability continues to increase post-adolescence into middle age: http://www.gnxp.com/blog/uploaded_images/mcgue-1993-increase-of-h2-decline-of-c2-755807.JPG
    "personality traits...would be influenced by IQ"

    I've heard that only the opennness to experience trait correlates with IQ, and that's because questions about creativity/opennness act as a self-report of intelligence.  But I've also heard of small correlations from other studies, so I'm not sure..

    ReplyDelete
  5. ben_g2:37 PM

    What are your thoughts about the Big Five model, as far as reliability and validity?

    ReplyDelete
  6. Wengchung3:01 PM

    > that's because questions about creativity/opennness act as a self-report of intelligence.

    I would have to disagree, according to my immaculate keystone source of wikipedia, openness to experience only correlates with creativity tests (30-40%) and crystallized intelligence (being general knowledge, vocabulary etc at 15-20%). The correlation of openness to fluid reasoning is negligible if even statistically significant at all. Based upon a mix of the correlations and heritability of openness as a single variable, it does seem like openness is probably measuring roughly the intended construct, not merely "how smart you think you are".

    It is interesting that openness (.5-.6, very little shared environment) and creativity tests (.2 heritability, uncertain of the dominance of shared environment) have practically the highest imaginable correlation between the two variables just based upon heritability, much higher than what one would expect, especially when considering that creativity tests only correlate together (on a subtest basis) about .07-.3.

    http://en.wikipedia.org/wiki/Openness_to_experience

    http://en.wikipedia.org/wiki/Big_Five_personality_traits

    Wikipedia sources for the heritability of openness and its correlations with other variables, you even get some fun sample questions!

    libres.uncg.edu/ir/uncg/f/P_Silvia_Openess_in%20press.pdfA pdf file containing another source for the correlation of openness to creativity.


    My source for the correlation of openness to tests of divergent thinking would be a paper by Richard Lynn in the mankind quarterly called "Race differences in Intelligence, Creativity, and Creative Achievement", in which he uses openness to experience as a rough metric for racial differences in creativity and attempts to connect them along with intelligence to historical accomplishment. Google the key words, there is a stolen version of the article uploaded on a webpage (the mankind quarterly requires a fee and may look weird on my credit history, I like to keep my evil and manic white supremacy ideals to myself). Lynn's conclusions in that paper are very controversial, so I'm going to refrain from linking it here.

    But about your other point upon psychological maturity between genders, that's interesting, thanks.

    ReplyDelete
  7. Wengchung3:05 PM

    Oh, another thing to note that I found kind of funny in the pdf file, openness to experience correlates much more strongly to creative achievement than the actual creativity tests. It could be considered "a more valid" measure of creativity.

    ReplyDelete
  8. MtMoru5:07 PM

    "We think GPA is a proxy for conscientiousness, or what is referred to below as grit."

    That is, it is not only a proxy, but it is a proxy for something which does not exist. Wunderbar!

    "Instead we use proxies for grit, like extracurriculars."

    "We being American colleges and unis and NO ONE else.

    "zest, grit, self-control, social intelligence, gratitude, optimism and curiosity."

    Looking for explanations of social outcomes, formal education perfromance in Les Etats-Unis Merdeux, in individual traits. In collectivist vs individualist cultures and Mediterranean vs Protestant cultures there should be less of this NAKED ideology. Except the "researchers" are totally unaware that they are naked.

    Steve should stick to science and stop trying to think, as Nassim Taleb might say. 

    ReplyDelete
  9. Guy_Brodude5:11 PM

    I'm most interested in the predictive power of subject-specific tests (e.g. SAT II's, AP exams, IB) as a measure of conscientiousness. Seems like those would bridge the gap nicely between the heavily g-loaded SAT and high school GPA, which is very difficult to compare across schools.

    ReplyDelete
  10. MtMoru5:14 PM

    They are more reliable the lower the IQ. If you've ever tired to fill out a personality test you, smart ben_g, should have found the questions so vague thet they were impossible to answer. So what happens is the less intelligent answer anyway but not according to reflection on their own behavior.

    ReplyDelete
  11. MtMoru5:15 PM

    Girls do what they're told.

    ReplyDelete
  12. MtMoru5:16 PM

    "maturity" doesn't mean anything. Let's just argue about the meaning of wrods.

    ReplyDelete
  13. ben_g7:51 PM

    People have different personality's depending on the situation they're in.  You have a personality for school, a personality for family, even a personality for making comments on websites. Of course, all these personas correlate with one another because you bring your genes with you everywhere you go.

    The most important thing to measure, for Steve's purposes, is the "school personality".  A well supported finding in psychology is that even minor adjustments to the testing environment can have huge effects.  If children are given tests around their family, the family personality comes out on the test.  If they take the test in a room with a whiteboard and lots of desks, the school personality comes out..

    Steve will administer these tests online, pretty much the same way he would send out a homework assignment.  That might prime people pretty well for their school personality

    ReplyDelete
  14. ben_g7:59 PM

    For those who want to see what personality questionairres look like, here's an index of open source ones: http://ipip.ori.org/newIndexofScaleLabels.htm

    ReplyDelete
  15. Wengchung10:14 PM

    >Tend to vote for liberal political candidates.

    This is listed under the "openness to experience" example questions.

    I facepalmed, grunted, then began furiously typing this.

    ReplyDelete
  16. MtMoru12:49 AM

    "Of course, all these personas correlate with one another because you bring your genes with you everywhere you go."

    Is that an example of the synthetic a priori or idoelogy?

    ReplyDelete
  17. MtMoru12:56 AM

    "men mature later"

    Ha! What does that even mean?

    It may be cultural but when I think if un-serious, frivolous people I think of women.

    A better way to say "mature later" is "are domesticated later". Civilized and especially industiralized man is a domesticated animal. I remember when I was in college there would be a huge crowd pouring out of a lecture hall and I would think, "Mooooooooooo!'

    ReplyDelete
  18. ben_g3:38 AM

    I don't know Kantian philosophy, but based on some wiki, I'd say it's an example of a synthetic posteriori.. Basically, multivariate genetic studies find that the correlation between personality tests taken in different settings is best explained by a model where the correlation is caused by genetic rather than environmental variation.  

    The multivariate stuff uses the same methods as the twin studies but with N number of traits-- fit a model to the correlations between people at different levels of relatedness.

    ReplyDelete
  19. whatisgoingon whatisgoingon12:35 PM

    While I believe the bottom part can be explained by mens laziness, I am not sure the top half can be explained by women really studying more in school. I believe it may simply be explained by differences in college major.

    For instance, using a common conversion, a 2.9 gpa in male dominated engineering is about a 3.3 in several female dominated social science courses.  And, while females applying to med school have higher non-science GPA's, men have higher science GPA's.

    So I believe the top portion really is just a statistical artifact.

    ReplyDelete
  20. IIRC the effect is still there when you look major by major.

    ReplyDelete
  21. LaurentMelchiorTellier3:52 PM

    Interesting article on data mining your students.

    Might it not also be the case that some universities (classes, countries, school districts, etc) will average higher levels of grit in their students, as a partial reflection of "goodness of school"? Some universities (courses, teachers, etc) might produce more stars in the upper left than others, despite not having the highest SAT / IQ students.
    This measure of goodness might avoid the "no child left behind" tarpit of making the school want to avoid low-SAT students who will pull more traditional "goodness" measures down, and of not putting effort into improving the gifted kids who will breeze through class anyway.(caveat 1: I'm not sure how GPA is worked out in the US, but I'm assuming it's on standardized tests for all universities, like in Denmark - otherwise, we're just measuring which schools are generous with grades, obviously)
    (caveat 2: this measure of goodness would probably infuriate somebody)

    ReplyDelete
  22. Yes, one could try to measure "value added" by the school. One proposal (see Hanushek at Stanford) is to measure value added for individual K-12 teachers by looking at improvement in scores over the course of a year. In the same way, if you had an objective measure of learning (perhaps the GRE subject test), you could measure how well graduates of a particular school perform relative to SAT score, and perhaps attribute the residual to the school. In the US GPAs are not standardized across universities.

    ReplyDelete
  23. Guy_Brodude4:26 PM

    With regards to your first caveat, GPA is based on marks awarded by the teachers of the student's high school courses. This is the single most important factor in college admissions in the US. The significant variance in difficulty and rigor between US high schools has been noted by many, and is often a sore spot for those who attend these rigorous schools (although the most rigorous schools still send the most applicants to competitive universities, so you might argue that it all evens out). The only justification for this system, as far as I can tell, is that it gives admissions committees a plausible excuse to admit under-qualified minority, athlete, and to a far lesser extent, economically-disadvantaged students who generally cannot post competitive scores on standardized examinations.

    Ostensibly GPA is a good measure of conscientiousness, but as I said in my earlier post I think subject-specific exams have higher predictive power.

    ReplyDelete
  24. LaurentMelchiorTellier4:29 PM

    I'd have to agree. Darn, that makes GPA a very noisy signal.

    ReplyDelete
  25. Guy_Brodude4:49 PM

    "Using GPA as a component in a goodness measure would only incentivize teachers to be more generous."

    Indeed. I would go so far as to say that a lot of the problems we have in this country with grade inflation (both at the secondary and post-secondary levels) are directly related to the use of school GPA as an admissions metric (it takes on a less important, but still very significant role in admissions to law, medical and business schools). Teachers feel guilty about giving "punitive" grades (anything below a 'B'), and students are thus rewarded for sub-standard work. How can this not have a negative effect on real acquisition and retention of knowledge?

    ReplyDelete
  26. MtMoru5:04 PM

    "I'm not sure how GPA is worked out in the US, but I'm assuming it's on standardized tests for all universities, like in Denmark"

    Oh no. In the Us multiple grades are given every semester (half year) or quarter and are determined by multiple tests (which are not standardized) and multiple "assignments". Testsd and assignments vary from one university to another and from one professor to another within each university.

    One of the results of this is that the US elite is much duller, much more obedient, and much pushier than that of any other rich country. For example, this blog's author will defend this way of doing things because he is himself unimaginaitive and obedient.

    ReplyDelete
  27. MtMoru5:14 PM

    "Ostensibly GPA is a good measure of conscientiousness"

    The problem is "consientiousness" is a virtue, and thus getting good grades is a virtue.

    But getting good grades can also be the result of vices. Obedience, pushiness, and domestication for example.

    Good grades are also more likely for those with the vice of extroversion in Jung's sense, that is the tendency to take as good, beautiful, important, etc. whatever ever one's immediate society or crowd takes as good, beautiful, important, etc. rather than determining these for oneself. Without extroversion in this sense there is no group think.

    ReplyDelete
  28. MtMoru7:59 PM

    What if there were clones of all the people in the lower right and upper left and these clones were spread around the world, some in Denmark, some in Brazil, etc.

    Would they still be the lower right and upper left people (using their countries' respective measures of academic ability and performance)?

    I say, definitely not. But Steve cannot even ask this question.

    ReplyDelete
  29. Imagine a logistic curve modeling probability of success and/or quality of work on the y-axis as a function of preexisting ability + effort on the x-axis.  If one gives assignments that are too easy or too difficult, final output is not very sensitive to the amount of effort put in.  On the other hand, if assignments are well-targeted to the individual's current ability, the return to effort is large and thus more effort will usually be made.

    To best cultivate grit, combine this with explicitly teaching what Carol Dweck calls a "growth mindset".

    As for the definition of "grit", I think of it as a combination of initiative and conscientiousness.  GPA (at least after controlling for institution and major) loads on the latter, but not so much on the former.  Historically, sports have been as good a predictor as any of initiative (hmm, how much of the variation can be explained by testosterone alone??), but I wonder how much damage modern helicopter parents have done to this signal.

    ReplyDelete
  30. LaurentMelchiorTellier10:57 PM

    Indeed. Growth mindset + a billion other factors, many of them environmental - as you say, what will happen when the kid gets out from under the heli-mom? And does that matter, if the heli-mom can afford to hover over him forever (or be replaced by a heli-girlfriend). What will happen when he gets sick, married, loses his lucky shorts, etc. And can any of this be called damage to the signal, or is it actually the genuine signal.

    *

    Gah, grit sounds like the kind of thing that's possible to quantify (in situ), but almost impossible to define/break down. Makes IQ seem downright inviting.

    ReplyDelete
  31. LaurentMelchiorTellier11:05 PM

    ^_^ 
    After discovering infoproc, I've been browsing your blog during my vacation, all the way back to some of your earliest posts. Suffice to say, one major reason I decided to pour some (modest) work into the cog study is precisely that you guys all strike me as good at considering imaginative ideas impartially and carefully.

    ReplyDelete
  32. whatisgoingon whatisgoingon4:02 AM

     http://www.aacom.org/data/Documents/Applicants/2009%20Osteopathic%20Medical%20School%20Applicants%209-1-2009.pdf

    This has quite a good deal of information that is relevant to your interests. It is all the data for the 2009 medical school applicants. 

    Something is odd in it though. Asians have lower science GPA's than whites, yet have much higher science MCAT scores, with the bulk of it being in the physical sciences(there is surprisingly little difference in the biology score)

    I am not quite sure of what to make of it. 

    As for gender, women have higher non-science GPA's, yet equal science GPA's. Since women get better grades for all sat scores, could the lower sci gpa be due to doing worse on math and physics(with the greater grit not being enough)  But than why would asians have lower sci gpa's? 

    Something seems very odd.

    ReplyDelete
  33. whatisgoingon whatisgoingon4:11 AM

    So, the the asian-white physical science difference section is very close to the male-female difference.

    I find that interesting, considering that other data has shown that the asian-white spatial IQ is similar to the male-female spatial IQ difference.

    ReplyDelete
  34. whatisgoingon whatisgoingon2:35 PM

    Oh I see.  This was osteopathic medicine(DO). For those applying to the more prestigious MD asians still do better in the science gpa, and have a similar difference(with an increase in the bio difference, yet not equal to the phy sci difference)

    ReplyDelete
  35. lovehorrorfilms6:51 PM

    "In previous work we found a .4 or so correlation between SAT score and upper division in-major GPA."

    The correlation would probably be much higher if you adjusted for restriction of range.  I imagine that people in the same college and same major have similar SATs so there's a limit on how well SAT can predict grades.  I've read that in elementary school, where the IQ range is much broader, IQ correlates 0.65 with school grades.   

    ReplyDelete
  36. Wengchung8:38 PM

    >Trying to explain social outcomes with traits of individuals when
    societies vary over space/place and time is pseudoscience.

    >traits of individuals

    Psychometrics is more valid for depicting broad trends than for predicting individual outcomes, because of the degree of inherent inaccuracy when assessing individual traits (especially in personality and creativity) and the fact that extensive testing is required for proper assessment. Giving someone Ravens Matrices would not be an appropriate assessment of g.

    But, you seem to be projecting some mad belief in tabula rasa and free will, which is kind of ironic based upon your accusation of pseudo-science.

    The human psyche is very static and concrete after early age, shared environment has hardly any impact upon intelligence and personality and the rest of the observed "non-genetic" variance is caused by a mix of error or perhaps individual environment.

    Of course not all of one's life outcome is absolutely based upon genes, in the United States educational attainment is around 30-40% a product of shared environment and the selectivity of the school you attend is even higher (an inference based upon data from Korean twins about the selectivity).

    But nonetheless, strong correlations exist between intelligence, personality, and things we consider to be aspects of "success".

    In different societies and environments (assuming they are sane), the degree of variance may alter, but the population on the top corner of the chart and the one at the bottom would still remain.

    ReplyDelete
  37. MtMoru1:18 AM

    "But, you seem to be projecting some mad belief in tabula rasa"

    Is English your native language? You write a lot for someone with no clue.

    ReplyDelete
  38. MtMoru1:20 AM

    "As for the definition of "grit", I think of it as a combination of initiative and conscientiousness."

    That is a like to put words together which are totally meaningless rather than think.

    ReplyDelete
  39. MtMoru1:27 AM

    GOTCHA. It's so easy.

    ReplyDelete
  40. Jean Huiskamps4:27 AM

    At the moment I'm reading Karl Marx's biography. I conclude that he was anything but "gritty" - scores of interests, changing all the time, scores of book-projects, announnced, begun, never finished. And see what an impact he has made anyway!

    ReplyDelete
  41. statsquatch4:33 PM

    Duckworth is a hack.  Her re-branding of conscientiousness as grit is derivative and her psychometrics is sloppy and/or shady:http://statsquatch.blogspot.com/2011/05/bad-apple-in-duckworths-iq-motivation.html.   

    ReplyDelete
  42. Much of the difference in male/female conscientiousness is probably explained by evolutionary incentives.  Women are rewarded more for conformity, and men for non-conformity.  Women are rewarded more for modest performance, men live in a sink or swim winner take all model for status and mating opportunities.  The truth is minor increases in income/status that come with conscientiousness in school seem to be a stronger incentive for women then men.  Getting a job that pays marginally more or has marginally better status just due to having slightly higher grades isn't worth the hassle for most guys, because marginal improvements don't enhance their happiness as much.

    The smartest most driven guys seem to be drop outs or people who tried to spend as little time as possible dealing with school so they can work on important projects.  The mediocre ones seem the most worried about doing well in school, since they sense they won't make it in the world on their own.  At the high end many probably realize that perfect 4.0s can be a turn off for some employers.  You'd be happier and better off diversifying away from school a bit.

    Simply put, if your getting a 3.X its hard to see how working your butt off to get a 3.X+0.25 is going to change your life outcome significantly. 

    ReplyDelete
  43. Tangentially related to the topic of test scores and colleges and the
    likes, but the recent brouhaha over the affirmative action bake sale at
    Berkeley simultaneously amused and annoyed me. So apparently to protest
    affirmative action, the Berkeley Republicans decided to charge whites
    $2.00 for a cupcake, Asians $1.50, Latinos $1.00 and blacks $.75. Now
    that's incredibly funny, because as I seem to recall Asian Americans
    were the most discriminated against group in college admissions. And if I
    also recall correctly, not only did whites benefit from affirmative
    action in the form of legacy and athletic admissions, but according to
    Espenshade, even when matched for the relevant non-academic criteria,
    Asian Americans still suffered a discriminatory penalty.



    Since white Americans are so vocal about protesting affirmative action,
    in this case under the mistaken belief that they're the ones hurt the
    most by AA, I wonder if Asian Americans will ever become as vocal in
    protesting this kind of thing. I'd love to see an Asian American group
    on a prominent campus start their own affirmative action bake sale
    protesting anti-Asian racism in college admissions. I know that Steve
    has often been interviewed for articles related to Asian Americans and
    AA and I'm not sure how he feels, but I find it incredibly annoying to
    hear white Americans complain about how AA hurts them the most when in
    fact they benefit from it relative to Asian Americans. This goes back to
    a key theme I addressed before, the marginalization of Asian Americans
    by both the white left and the white right. The white left wants to
    maintain the idea that white privilege hurts blacks and Hispanics and in
    order to do so they have to de-emphasize Asian American academic
    achievement relative to whites and ignore the painful fact that Asian
    Americans are really the most discriminated against groups. The white
    right on the other hand is basically a bunch of oblivious "angry white
    male" types, totally out of touch with empirical reality when it comes
    to the college admissions process.

    ReplyDelete
  44. ben_g4:23 PM

    Yan,

    I was also surprised at how the pricing scheme put whites above asians. but if you look at the berkeley college republicans web page you'll see that about half of them are asian and one is hispanic. Apparently the discimination against asians is something most people are ignorant of regardless of race

    ReplyDelete