The value of hard work

Our scheduled departmental colloquium was cancelled this week, so my colleague Jim Schombert and I volunteered to talk about some work we're doing on the predictive power of the SAT. The talk is Thursday, Feb. 11, 2010 at 4 PM.

When I get my collaborator's approval I will post a link to all sorts of fun graphs. Here are some introductory slides I prepared to explain a bit about psychometrics to physicists. [Graphs are now available!]

Overall, the message is hopeful: SAT score only accounts for a fraction of total variation in college success. Other factors, such as hard work or conscientiousness, probably play at least as large a role. Nevertheless, the SAT has clear (statistical) predictive power -- about as much as high school GPA. One caveat to the abstract below is that "overachievers" (as defined) tend to be concentrated in certain majors (in rough order of prevalence: sociology, political science, humanities, biology, chemistry); they are harder to find in subjects like pure math, rigorous computer science and physics, which seem to have actual cognitive thresholds.

Jim Schombert and Steve Hsu will discuss some statistical research on SAT scores and UO grades based on a large corpus of student data. After a brief discussion of psychometrics (cognitive testing, the meaning of the SAT and GRE), the authors will discuss the points outlined below.


Title: The Value of Hard Work: College GPA Predictions From SAT Scores

Abstract:

We analyze a data set composed of the academic records of all undergraduates entering the University of Oregon from 2000-2004. We find correlations of roughly .3 to .5 between SAT scores and upper division, in-major GPA (henceforth, GPA). Interestingly, low SAT scores do not preclude high performance in most majors. That is, the distribution of SAT scores after conditioning on high GPA (e.g., > 3.5 or even 4.0) typically extends below 1000 (the average among test takers). We hypothesize that overachievers overcome cognitive deficits through hard work, and discuss to what extent they can be identified from high school records. Only a few majors seem to exhibit a cognitive threshold -- i.e., such that high GPA (mastery of the subject matter) is very unlikely below a certain SAT threshold (i.e., no matter how dedicated or hard working the student). Our results suggest that almost any student admitted to university can achieve academic success, if they work hard enough.

We find that the best predictor of GPA is a roughly equally weighted sum of SAT and high school GPA, measured in standard deviation units. We also analyze the performance of UO honors college students, a selected population which resembles that of elite private colleges. Finally, we observe that 1. SAT scores fluctuate little on retest (very high reliability), 2. SAT and GRE scores (where available) correlate at roughly .75, consistent with the notion that both tests measure a relatively stable general cognitive ability, and 3. the SAT distribution of students that obtained a degree does not differ substantially from that of the entering class.

Below is a graph showing the reliability of SAT scores. It gives the frequency of score differences (max minus avg or max minus min) for students who took the test more than once. The result for verbal (reading) scores is about the same. Improvements of more than 1 SD (100 points) are quite rare. It seems likely that among the thousands of students in this data set, at least a few used SAT prep courses, but apparently with limited success.




A partial list of the graphs available here. Note UO GPA is always in-major, upper division GPA. In the case of math and CIS (computer science) the grades are from a subset of especially rigorous courses in each department.

SAT combined vs UO GPA by major
SAT-M,V vs UO GPA by major
HSGPA vs UO GPA by major
SAT combined vs HS GPA

Other plots:
SAT profiles of graduates and non-graduates
retest reliability of SAT M, V
SAT-M,V and GRE correlation
Clark Honors College GPA and SAT vs overall UO population
UO GPA vs best (equal weight) SAT + HSGPA predictor

Clark Honors College (CHC) students are roughly equivalent to Berkeley or Cornell students, based on SAT and HSGPA. They outperform typical UO students, but you can see that even CHC GPAs (again, in-major, upper div) cover a wide range. So, it seems likely that UO students with high in-major GPAs have subject mastery similar to the better students at elite universities. The CHC students are the red dots in the graph below.




Here is UO GPA vs best predictor: equally weighted sum of SAT and high school GPA, measured in standard deviation units.

The measure of success

This article in the Oregon Quarterly (alumni magazine) describes my research with colleague Jim Schombert on college GPA predictions from SAT scores.

“Freshman GPA is not a satisfactory metric of academic success,” Hsu explains. “There is simply too much variation in the difficulty of courses taken by freshmen.” More able freshmen typically take more difficult courses, whereas less able freshmen take introductory courses “not very different from high school classes,” he says. Under these circumstances, academic success—an “A” in an introductory course versus a “B” in an advanced course—becomes too relative to accurately measure. Course variation decreases in later years, as students settle into their respective majors, working hard in required classes.

The new approach bore fruit: SAT and ACT scores, their analysis showed, predict upper-level much better than lower-level college grades, “a significant and entirely new result,” Schombert says.

To be precise, the correlation between SAT and GPA in upper division, in-major courses (presumably the most important courses in a particular student's college career) is much higher than the often reported correlation with freshman GPA.

The feel-good conclusion from our work is that in most majors (e.g., History, English, Biology, social sciences, ...) students with modest SAT scores can still obtain high GPAs, presumably through hard work. However, we found almost no cases of SAT-M scorers below about 90th percentile who obtained high upper division GPAs in physics or pure mathematics (second link below).

Related posts:

Data mining the university , Psychometric thresholds for physics and mathematics


I was invited to participate in an August 26 WGBH radio show on Affirmative Action, but couldn't because it coincided with my travel to Taiwan. (You have to search around a bit at the link for the audio; AA discussion starts about 12 minutes in.) Really a pity because the participants got caught up on the issue of whether SAT is a decent measure of academic ability, with Lani Guinier and Oiyan Poon both asserting the (incorrect) claim that it is not, based on low correlation between SAT and freshman GPA at schools with a significant restriction of range.

Affirmative Action In Education: We’ll discuss the merits and pitfalls of affirmative action, its broader implications in the admissions process, and the untold story of those not just being excluded – but indirectly penalized in the process. We’re joined by Dan Golden, author of The Price of Admission; Harvey Mansfield, a political philosophy and government professor at Harvard University; Oiyan A. Poon, a research associate at UMass Boston’s Institute for Asian American Studies; and Harvard Law School Professor Lani Guinier.

Here's my email to Kara Miller, who hosted the radio segment.

Kara,

I'm really sorry I missed participating in the program, because I've recently done some research on exactly the question that occupied a central place in the heated discussion that ended your show.

It turns out Mansfield is correct -- SATs and ACTs do predict college performance. In fact, in fields like physics and mathematics there appears to be a hard threshold -- students below 90th percentile or so on standardized tests of math ability almost never do well enough to be admitted to a PhD program, no matter how hard they work. Also, probability of success in mastering the undergraduate curriculum in these fields rises rapidly with SAT math score. Mansfield is right to distrust the studies on this topic by scholars in education or social science. My own analysis of 10 years of detailed U Oregon records yields very different conclusions than the ones cited by Guinier and Poon (see below). I can explain why their conclusions are erroneous, but not without discussing statistical issues that require a high SAT math score to understand ;-)

Best,
Steve

Data Mining the University

Here is a draft of my paper with Jim Schombert on University of Oregon GPA and SAT statistics. I posted previously on this research: Cognitive thresholds , The value of hard work. Introductory slides on g, SAT and all that.

Much of our data is available in the plots here.

Why did I get interested in this stuff? Obviously, I have a long-standing interest in psychometrics. In teaching 100 level courses, both Schombert and I have been flummoxed at the large population of students who have trouble with what we would consider elementary concepts (e.g., "scaling", or even "area" or "volume"!), yet seem to be successful in their chosen major ("I just can't seem to do these problems, but I need this class to graduate (fulfill a science requirement). I always get A's in English/History/Sociology/ ..."!) I'm sure every physics professor hears these things. Because I invest a lot of time in helping students, e.g., solving lots of problems during office hours, I have a pretty close view of their learning abilities. I began to wonder how students could have trouble with these basic concepts ("Didn't you have to know that for the SAT?"), yet have high GPAs in their major. In talking to Jim, who is director of the General Science program, we realized the data was actually available to investigate these questions further.

Data Mining the University: College GPA Predictions from SAT Scores


From the Conclusions:

1. SATs predict upper GPA with correlations in the 0.35 -- 0.50
range.

2. Overachievers exist in most majors, with low SAT scores but
very high GPAs. These overachievers are disproportionately female.

3. Underachievers exist in all majors, with high SAT scores but
very low GPAs. These underachievers are disproportionately male.

4. Some majors, like math and physics, may exhibit a cognitive threshold -- mastery of the material is unlikely below an ability threshold (as measured by SAT-M), no matter how hard the student works.

5. Students at public universities, like UO, with high upper GPA (e.g., 3.7 or greater) likely have subject mastery similar to graduates of elite universities. Elite college students who transferred to a state university would likely average upper division GPAs of 3.7 or greater.

Figure 8 caption: Underachievers and Overachievers (red = females, blue = males) isolated by SAT and upper GPA (1.25 standard deviations from the green ridgeline). The overachievers are mostly female students (64%) and the underachievers are mostly male students (79%). (Click for larger version.)

True grit

The next project with my colleague Jim Schombert is to see whether student personality inventories increase our ability to predict college GPA. In previous work we found a .4 or so correlation between SAT score and upper division in-major GPA. (Other studies, which focus on freshman GPA, typically find lower correlations but this is partly because academically stronger freshmen usually take harder courses. At the upper division level, majors typically have to take certain core courses, so there is more uniformity.) The correlation is somewhat higher (.5 to .6) if we use a z score derived from high school GPA and SAT. We think GPA is a proxy for conscientiousness, or what is referred to below as grit. But there is too much grade inflation in high school these days, and GPA depends both on work ethic and cognitive ability. So we'd like to see how well personality variables work. Optimistically, I think we can do better than correlations of .6, which is pretty impressive for social science.

NYTimes: ... People who accomplished great things, she noticed, often combined a passion for a single mission with an unswerving dedication to achieve that mission, whatever the obstacles and however long it might take. She decided she needed to name this quality, and she chose the word “grit.”

She developed a test to measure grit, which she called the Grit Scale. It is a deceptively simple test, in that it requires you to rate yourself on just 12 questions, from “I finish whatever I begin” to “I often set a goal but later choose to pursue a different one.” It takes about three minutes to complete, and it relies entirely on self-report — and yet when Duckworth took it out into the field, she found it was remarkably predictive of success. At Penn, high grit ratings allowed students with relatively low college-board scores to nonetheless achieve high G.P.A.’s. Duckworth and her collaborators gave their grit test to more than 1,200 freshman cadets as they entered West Point and embarked on the grueling summer training course known as Beast Barracks. The military has developed its own complex evaluation, called the Whole Candidate Score, to judge incoming cadets and predict which of them will survive the demands of West Point; it includes academic grades, a gauge of physical fitness and a Leadership Potential Score. But at the end of Beast Barracks, the more accurate predictor of which cadets persisted and which ones dropped out turned out to be Duckworth’s 12-item grit questionnaire.

[Of course, the grit score can't be used for admissions -- too easy to game, unlike an IQ test! Instead we use proxies for grit, like extracurriculars.]

... The first question Duckworth addressed, again, was the relative importance of I.Q. and self-control. She and her team of researchers gave middle-school students at Riverdale and KIPP a variety of psychological and I.Q. tests. They found that at both schools, I.Q. was the better predictor of scores on statewide achievement tests, but measures of self-control were more reliable indicators of report-card grades.

Duckworth’s research convinced Levin and Randolph that they should try to foster self-control and grit in their students. Yet those didn’t seem like the only character strengths that mattered. ... After a few small adjustments (Levin and Randolph opted to drop love in favor of curiosity), they settled on a final list: zest, grit, self-control, social intelligence, gratitude, optimism and curiosity.

Here's a figure from our paper Data Mining the University, which shows how SAT predicts GPA. The stars are over- and under-achievers (blue = male, red = female). Do the people in the upper left (over-achievers) have grit?

Rigorous inequalities

The Effects of an Anti-grade-Inflation Policy at Wellesley College
Journal of Economic Perspectives, 28(3): 189-204 (2014)
DOI: 10.1257/jep.28.3.189

Average grades in colleges and universities have risen markedly since the 1960s. Critics express concern that grade inflation erodes incentives for students to learn; gives students, employers, and graduate schools poor information on absolute and relative abilities; and reflects the quid pro quo of grades for better student evaluations of professors. This paper evaluates an anti-grade-inflation policy that capped most course averages at a B+. The cap was biding for high-grading departments (in the humanities and social sciences) and was not binding for low-grading departments (in economics and sciences), facilitating a difference-in-differences analysis. Professors complied with the policy by reducing compression at the top of the grade distribution. It had little effect on receipt of top honors, but affected receipt of magna cum laude. In departments affected by the cap, the policy expanded racial gaps in grades, reduced enrollments and majors, and lowered student ratings of professors.

Jim Schombert and I discovered similar disparities in our study of University of Oregon student grades. The inequities would be even larger after controlling for student ability. Eventually employers may demand learning outcomes testing (see Measuring college learning outcomes: psychometry 101), and the results won't be pretty.

Via Carl Shulman and orgtheory.

Psychometric thresholds for physics and mathematics

This is a follow up to our earlier paper on GPA-SAT correlations. Click below for the pdf.

Non-linear Psychometric Thresholds for Physics and Mathematics

Stephen D.H. Hsu, James Schombert

We analyze 5 years of student records at the University of Oregon to estimate the probability of success (as defined by superior undergraduate record; sufficient for admission to graduate school) in Physics and Mathematics as a function of SAT-M score. We find evidence of a non-linear threshold: below SAT-M score of roughly 600, the probability of success is very low. Interestingly, no similar threshold exists in other majors, such as Sociology, History, English or Biology, whether on SAT combined, SAT-R or SAT-M. Our findings have significant implications for the demographic makeup of graduate populations in mathematically intensive subjects, given the current distribution of SAT-M scores.

There is clearly something different about the physics and math GPA vs SAT distributions compared to all of the other majors we looked at (see figure 1 in the paper). In the other majors (history, sociology, etc.) it appears that hard work can compensate for low SAT score. But that is not the case in math and physics.

One interesting question is whether the apparent cognitive threshold is a linear or non-linear effect. Our data suggests that the probability of doing well in any particular quarter of introductory physics may be linear with SAT-M, but the probability of having a high cumulative GPA in physics or math is very non-linear in SAT-M. See figure below: the red line is the upper bound at 95% confidence level on the probability of getting an A in a particular quarter of introductory physics, and the blue line is an upper bound on the probability of earning a cumulative GPA of at least 3.5 or so. The central values of these probabilities are much lower than the 95% confidence upper bounds: to have a 50% chance of GPA > 3.5 requires SAT-M in the top few percent of the population, judging just by the distribution of individuals in our sample. The confidence intervals are larger in the tails because the number of individuals is small.

Fig. 4.— Probability charts for SAT Math versus grades in Physics classes. For each SAT bin, the number of A-type grades (n) is listed along with the total number of grades per bin (N). The red line displays the 95% probability P(95%) and the blue symbols display an upper bound on the probability that a student will achieve 8 or more A’s out of 16 courses (typically necessary for an upper division GPA of 3.5.) Click image for better version.