When I get my collaborator's approval I will post a link to all sorts of fun graphs. Here are some introductory slides I prepared to explain a bit about psychometrics to physicists. [Graphs are now available!]
Overall, the message is hopeful: SAT score only accounts for a fraction of total variation in college success. Other factors, such as hard work or conscientiousness, probably play at least as large a role. Nevertheless, the SAT has clear (statistical) predictive power -- about as much as high school GPA. One caveat to the abstract below is that "overachievers" (as defined) tend to be concentrated in certain majors (in rough order of prevalence: sociology, political science, humanities, biology, chemistry); they are harder to find in subjects like pure math, rigorous computer science and physics, which seem to have actual cognitive thresholds.
Jim Schombert and Steve Hsu will discuss some statistical research on SAT scores and UO grades based on a large corpus of student data. After a brief discussion of psychometrics (cognitive testing, the meaning of the SAT and GRE), the authors will discuss the points outlined below.
Title: The Value of Hard Work: College GPA Predictions From SAT Scores
Abstract:
We analyze a data set composed of the academic records of all undergraduates entering the University of Oregon from 2000-2004. We find correlations of roughly .3 to .5 between SAT scores and upper division, in-major GPA (henceforth, GPA). Interestingly, low SAT scores do not preclude high performance in most majors. That is, the distribution of SAT scores after conditioning on high GPA (e.g., > 3.5 or even 4.0) typically extends below 1000 (the average among test takers). We hypothesize that overachievers overcome cognitive deficits through hard work, and discuss to what extent they can be identified from high school records. Only a few majors seem to exhibit a cognitive threshold -- i.e., such that high GPA (mastery of the subject matter) is very unlikely below a certain SAT threshold (i.e., no matter how dedicated or hard working the student). Our results suggest that almost any student admitted to university can achieve academic success, if they work hard enough.
We find that the best predictor of GPA is a roughly equally weighted sum of SAT and high school GPA, measured in standard deviation units. We also analyze the performance of UO honors college students, a selected population which resembles that of elite private colleges. Finally, we observe that 1. SAT scores fluctuate little on retest (very high reliability), 2. SAT and GRE scores (where available) correlate at roughly .75, consistent with the notion that both tests measure a relatively stable general cognitive ability, and 3. the SAT distribution of students that obtained a degree does not differ substantially from that of the entering class.
Below is a graph showing the reliability of SAT scores. It gives the frequency of score differences (max minus avg or max minus min) for students who took the test more than once. The result for verbal (reading) scores is about the same. Improvements of more than 1 SD (100 points) are quite rare. It seems likely that among the thousands of students in this data set, at least a few used SAT prep courses, but apparently with limited success.
A partial list of the graphs available here. Note UO GPA is always in-major, upper division GPA. In the case of math and CIS (computer science) the grades are from a subset of especially rigorous courses in each department.
SAT combined vs UO GPA by major
SAT-M,V vs UO GPA by major
HSGPA vs UO GPA by major
SAT combined vs HS GPA
Other plots:
SAT profiles of graduates and non-graduates
retest reliability of SAT M, V
SAT-M,V and GRE correlation
Clark Honors College GPA and SAT vs overall UO population
UO GPA vs best (equal weight) SAT + HSGPA predictor
Clark Honors College (CHC) students are roughly equivalent to Berkeley or Cornell students, based on SAT and HSGPA. They outperform typical UO students, but you can see that even CHC GPAs (again, in-major, upper div) cover a wide range. So, it seems likely that UO students with high in-major GPAs have subject mastery similar to the better students at elite universities. The CHC students are the red dots in the graph below.
Here is UO GPA vs best predictor: equally weighted sum of SAT and high school GPA, measured in standard deviation units.
