The measure of success

This article in the Oregon Quarterly (alumni magazine) describes my research with colleague Jim Schombert on college GPA predictions from SAT scores.

“Freshman GPA is not a satisfactory metric of academic success,” Hsu explains. “There is simply too much variation in the difficulty of courses taken by freshmen.” More able freshmen typically take more difficult courses, whereas less able freshmen take introductory courses “not very different from high school classes,” he says. Under these circumstances, academic success—an “A” in an introductory course versus a “B” in an advanced course—becomes too relative to accurately measure. Course variation decreases in later years, as students settle into their respective majors, working hard in required classes.

The new approach bore fruit: SAT and ACT scores, their analysis showed, predict upper-level much better than lower-level college grades, “a significant and entirely new result,” Schombert says.

To be precise, the correlation between SAT and GPA in upper division, in-major courses (presumably the most important courses in a particular student's college career) is much higher than the often reported correlation with freshman GPA.

The feel-good conclusion from our work is that in most majors (e.g., History, English, Biology, social sciences, ...) students with modest SAT scores can still obtain high GPAs, presumably through hard work. However, we found almost no cases of SAT-M scorers below about 90th percentile who obtained high upper division GPAs in physics or pure mathematics (second link below).

Related posts:

Data mining the university , Psychometric thresholds for physics and mathematics


I was invited to participate in an August 26 WGBH radio show on Affirmative Action, but couldn't because it coincided with my travel to Taiwan. (You have to search around a bit at the link for the audio; AA discussion starts about 12 minutes in.) Really a pity because the participants got caught up on the issue of whether SAT is a decent measure of academic ability, with Lani Guinier and Oiyan Poon both asserting the (incorrect) claim that it is not, based on low correlation between SAT and freshman GPA at schools with a significant restriction of range.

Affirmative Action In Education: We’ll discuss the merits and pitfalls of affirmative action, its broader implications in the admissions process, and the untold story of those not just being excluded – but indirectly penalized in the process. We’re joined by Dan Golden, author of The Price of Admission; Harvey Mansfield, a political philosophy and government professor at Harvard University; Oiyan A. Poon, a research associate at UMass Boston’s Institute for Asian American Studies; and Harvard Law School Professor Lani Guinier.

Here's my email to Kara Miller, who hosted the radio segment.

Kara,

I'm really sorry I missed participating in the program, because I've recently done some research on exactly the question that occupied a central place in the heated discussion that ended your show.

It turns out Mansfield is correct -- SATs and ACTs do predict college performance. In fact, in fields like physics and mathematics there appears to be a hard threshold -- students below 90th percentile or so on standardized tests of math ability almost never do well enough to be admitted to a PhD program, no matter how hard they work. Also, probability of success in mastering the undergraduate curriculum in these fields rises rapidly with SAT math score. Mansfield is right to distrust the studies on this topic by scholars in education or social science. My own analysis of 10 years of detailed U Oregon records yields very different conclusions than the ones cited by Guinier and Poon (see below). I can explain why their conclusions are erroneous, but not without discussing statistical issues that require a high SAT math score to understand ;-)

Best,
Steve