When I get my collaborator's approval I will post a link to all sorts of fun graphs. Here are some introductory slides I prepared to explain a bit about psychometrics to physicists. [Graphs are now available!]
Overall, the message is hopeful: SAT score only accounts for a fraction of total variation in college success. Other factors, such as hard work or conscientiousness, probably play at least as large a role. Nevertheless, the SAT has clear (statistical) predictive power -- about as much as high school GPA. One caveat to the abstract below is that "overachievers" (as defined) tend to be concentrated in certain majors (in rough order of prevalence: sociology, political science, humanities, biology, chemistry); they are harder to find in subjects like pure math, rigorous computer science and physics, which seem to have actual cognitive thresholds.
Jim Schombert and Steve Hsu will discuss some statistical research on SAT scores and UO grades based on a large corpus of student data. After a brief discussion of psychometrics (cognitive testing, the meaning of the SAT and GRE), the authors will discuss the points outlined below.
Title: The Value of Hard Work: College GPA Predictions From SAT Scores
Abstract:
We analyze a data set composed of the academic records of all undergraduates entering the University of Oregon from 2000-2004. We find correlations of roughly .3 to .5 between SAT scores and upper division, in-major GPA (henceforth, GPA). Interestingly, low SAT scores do not preclude high performance in most majors. That is, the distribution of SAT scores after conditioning on high GPA (e.g., > 3.5 or even 4.0) typically extends below 1000 (the average among test takers). We hypothesize that overachievers overcome cognitive deficits through hard work, and discuss to what extent they can be identified from high school records. Only a few majors seem to exhibit a cognitive threshold -- i.e., such that high GPA (mastery of the subject matter) is very unlikely below a certain SAT threshold (i.e., no matter how dedicated or hard working the student). Our results suggest that almost any student admitted to university can achieve academic success, if they work hard enough.
We find that the best predictor of GPA is a roughly equally weighted sum of SAT and high school GPA, measured in standard deviation units. We also analyze the performance of UO honors college students, a selected population which resembles that of elite private colleges. Finally, we observe that 1. SAT scores fluctuate little on retest (very high reliability), 2. SAT and GRE scores (where available) correlate at roughly .75, consistent with the notion that both tests measure a relatively stable general cognitive ability, and 3. the SAT distribution of students that obtained a degree does not differ substantially from that of the entering class.
Below is a graph showing the reliability of SAT scores. It gives the frequency of score differences (max minus avg or max minus min) for students who took the test more than once. The result for verbal (reading) scores is about the same. Improvements of more than 1 SD (100 points) are quite rare. It seems likely that among the thousands of students in this data set, at least a few used SAT prep courses, but apparently with limited success.
A partial list of the graphs available here. Note UO GPA is always in-major, upper division GPA. In the case of math and CIS (computer science) the grades are from a subset of especially rigorous courses in each department.
SAT combined vs UO GPA by major
SAT-M,V vs UO GPA by major
HSGPA vs UO GPA by major
SAT combined vs HS GPA
Other plots:
SAT profiles of graduates and non-graduates
retest reliability of SAT M, V
SAT-M,V and GRE correlation
Clark Honors College GPA and SAT vs overall UO population
UO GPA vs best (equal weight) SAT + HSGPA predictor
Clark Honors College (CHC) students are roughly equivalent to Berkeley or Cornell students, based on SAT and HSGPA. They outperform typical UO students, but you can see that even CHC GPAs (again, in-major, upper div) cover a wide range. So, it seems likely that UO students with high in-major GPAs have subject mastery similar to the better students at elite universities. The CHC students are the red dots in the graph below.
Here is UO GPA vs best predictor: equally weighted sum of SAT and high school GPA, measured in standard deviation units.
Looking forward to more data!
ReplyDeleteThis comment has been removed by the author.
ReplyDelete"Other factors, such as hard work or conscientiousness, probably play at least as large a role"
ReplyDeletethat is ... PUSHINESS
"Overall, the message is hopeful: SAT score only accounts for a fraction of total variation in college success"
that is ... the opposite of hopeful
Death to America!
Would you be willing to share your raw data? I'm particularly interested in whether you found a cognitive threshold in philosophy like you found in pure math, etc..
ReplyDeleteIf you are, I'll send you a PM.
When I get the OK I'll post a link to a comprehensive set of graphs of our results.
ReplyDeleteTo answer your question, we don't see evidence of a cognitive threshold for the philosophy major. Philosophy students with SAT < 1000 and SAT-V < 600 or even 500 were able to obtain in-major, upper division GPAs of 3.5--4.0. The results are pretty similar to what we found for history majors, IIRC. It will be clear to you when you see the graphs.
PS Just to clarify, I didn't mean to say that the majority of low-SAT philosophy majors were able to obtain high GPAs, but rather that we have a nontrivial number of examples. The probability of this kind of "overachievement" seems lower in, e.g., physics or math, especially relative to SAT-M score.
ReplyDeletePhilosophy has zero market value.
ReplyDeletePhilosophy professors make a living off the hardship of their students during and after school.
They are rip-off artists/con men.
Don't be fooled.
This is real interesting, would love to see more. One question: have you given thought to selection?
ReplyDeleteThanks, Steve!
ReplyDeleteI once heard that the best thing a kids SAT predicted was the family's household income.
ReplyDeleteDo you have data on that?
Dave: there are a number of places where selection effects could be important. For example, the honors college kids have the SAT and GPA to be at elite privates, but perhaps there is some systematic effect related to them being here instead.
ReplyDeleteOne big selection effect that pertains to cognitive thresholds is that we only plot kids who actually graduated in the major. If you look at, e.g., physics, there might be a lot of kids who took the lower division classes but decided to switch majors before taking the upper div classes. They're not in our data set and if they were it might make the cognitive threshold question more transparent.
Martin: I'm sure there's an SAT-SES correlation. But there are poor kids with high SATs and rich kids with low SATs. I suspect it's the SAT score that is predictive, not the SES. We didn't look at that directly in our study but I'm sure it has been addressed elsewhere.
What a great blog chocerbloc full of interesting stuff! I see you are interested in statistical research. I have put one of the most comprehensive link lists for hundreds of thousands of statistical sources and indicators on my blog: Statistics Reference List. And what I find most fascinating is how data can be visualised nowadays with the graphical computing power of modern PCs, as in manyof the dozens of examples in these Data Visualisation References. If you miss anything that I might be able to find for you or you yourself want to share a resource, please leave a comment.
ReplyDeletePerhaps the Philosophy undergraduate program at UO is a bastion of grade inflation. There is no cognitive threshold because the professors/graders/etc choose to give good grades for mediocre work. This phenomenon can apply to many liberal arts major. It certainly happens at my University.
ReplyDeleteGPA in STEM fields and math SAT scores are another story.
The average grades in upper div humanities classes are only slightly higher in than in science classes. We've checked that the effect is not simply grade inflation. You can measure grades in terms of SD relative to mean for the major. The main point is that low SAT people are able to achieve GPAs well above the mean in humanities majors but not, e.g., in physics.
ReplyDeleteIs that why physics was the easiest subject for me?
ReplyDeleteAnd I had much more trouble with humanities courses despite a 760 SAT verbal?
DEATH TO AMERICA!
Interesting. I see your data, but what I am concerned about is your premise that high GPA is indicative of mastery of course material. What if it isn't? I'm no quant, but I still don't see how your research can prove that GPA actually shows mastery of the material.
ReplyDeleteYou also appear to assume that if a student with low SAT attains a high GPA, this means that they worked hard. What if they didn't work hard at all, and the classes are just really easy? Maybe if you find data on hours spent studying, it would help your case.
The claim that hard work makes the difference is speculative. Of course, it could be some other compensating factor. However, as someone with a fair amount of teaching experience (not to mention experience as a student), it's clear to me that effort can compensate for ability.
ReplyDeleteThe claim that a high grade is related to mastery is subject to similar uncertainty, although I imagine most professors would claim they are correlated. Certainly in physics you can tell easily from final exams what the level of mastery is.
"Certainly in physics you can tell easily from final exams what the level of mastery is."
ReplyDeleteThis is a cognitive bias of yours Steve. Grades do not correlate with SAT/IQ so much for reasons other than "conscientiousness".
Grades are 1) subjective 2) cover learning over a short period of time.
It is interesting that gpa is taken by universities and by you as the measure of success or failure.
In most other countries cummulative exams are the measure.
If you compared SAT to GRE subject test scores the correlation would be much higher than with gpa, I'm sure. The same for "cums" at the grad school level and GREs provided the cums are objective.
I am an example. I graduated in mathematics with a cummulative gpa below 3 in maths, SAT score = 1560.
But when I sat the old exam 100 of the SoA my score was highest of the 1665 people who took it on the same date.
I worked with a woman with a masters in stats from a Cali U (I could never have gotten in). She studied for the exam 100 intensely and failed it three times in a row.
Some day the US will pay a price for its (almost uniquely) inefficient allocation of human capital. Canada is the only country which is worse.
This comment has been removed by the author.
ReplyDeleteNational comprehensive subject exams would solve the problem, if only the powers that be had any interest in implementing. Both colleges and the government are too afraid of the potential results. What if are kids really didn't learn ANYTHING in college? That would be a shocker.
ReplyDelete^I agree with poster above. Steve is clearly affected by cognitive bias. Have you met anyone from the lower half of the bell curve? Tell them to try hard.
Physics exams are far more subjective than humanities exams.
A good measure would be comparing the SAT of philosophy students with both their GPAs AND their LSAT scores (if taken).
"Physics exams are far more subjective than humanities exams."
ReplyDelete^^^A comedian.
^
ReplyDeleteOops. I meant objective. I mixed it up.
Another issue I have with the study is the statement, "So, it seems likely that UO students with high in-major GPAs have subject mastery similar to the better students at elite universities."
This statement assumes that the course material and expectation levels at UO are the same as at a elite universities. When dealing with upper division courses (typically non-honors, specific subjects, this is how it is at most places, I'm not too sure about UO), I would expect that course material and expectations are lower, because you have non-honors students in the class as well, causing course material to be dumbed down/grading standards to be relaxed. I would expect this effect to be more concentrated in the humanities, less in the sciences.
At my large, highly regarded public university, upper division liberal arts classes are fairly easy, even for cognitively deficient students. They don't require a lot of studying time at all.
Victor Niederhoffer made at least a B by taking only graduate courses.
ReplyDeleteHe claims there is an expression "Niederhoffering", which means to take only grad courses to assure at least a B.