This graph (click for larger version) shows the upper division in-major GPAs of U Oregon physics and math graduates from a 5 year period. The math GPAs were computed using a specific set of rigorous courses taken by graduate school bound majors. One might reasonably associate "mastery" of a subject with GPA > 3.5 -- roughly, the minimum threshold to be admitted to graduate school (i.e., students who earn equal numbers of A's and B's in their upper division core courses are borderline for most graduate programs). We find that no student with SAT-M less than roughly 600 was able to attain this level of mastery.
This data provides at least modest evidence for a cognitive threshold required for physics and mathematics. That is, the a priori probability that a student with SAT-M < 600 will perform well enough to be admitted to graduate school in these subjects is extremely low. Note many poorly performing students switch majors, and hence do not populate the lower left corner of our graph.
A simple but plausible model for college performance might include two factors: 1. ability (measured by SAT) and 2. conscientiousness or effort (for simplicity, an uncorrelated random variable, probably normally distributed). The predicted GPA might depend linearly on the sum of the two factors, each measured in SD units. (See bottom figure here for a similar predictor of GPA.) Applying such a model to our data, we would conclude that even a student in the, e.g., 90th percentile of work ethic has a low probability of attaining mastery if their SAT-M score is below 600.
To reiterate, SAT-M 600 seems to be the lowest score at which even a very motivated student has a chance for mastery. From the data one might guess that only for SAT-M well above 700 do students have more than a 50 percent chance of obtaining GPA > 3.5. That is, a student with average motivation or conscientiousness probably needs SAT-M well above 700 to have a high probability of obtaining mastery.
We were unable to find any similar threshold (either in SAT-V or SAT-M) in other majors, including economics, sociology, history, philosophy, biology, chemistry, etc. (More about this data set, including data on other majors, here.)
The percentile equivalents of SAT-M scores for various groups can be found here. For the total SAT-taking population, 600 is about 75th percentile. For the overall population, it might be 85th percentile. SAT-M of 750 is roughly 98th percentile for the total SAT-taking population.
For comparison, here is the analogous plot for English (black) and History (blue) majors versus SAT-R (verbal). If a threshold exists it is probably at SAT-R of 450 or so.
The graph doesn't seem to support your assertions. E.g., a few points appear to show GPA above 3.5, but SAT less than 600. Are the axes right?
ReplyDeleteYou don't have enough data.
ReplyDeleteThe cognitive threshold may well exist, but, based on data at hand, it appears to me that you can only set modest limits on its existence, saying something along the lines of "... confidence level of the probability for a SAT-M 550 student to succeed in UO is 0% to 40% @ 95% CL, as contrasted with the confidence level of 30% to 80% for a SAT-M 750 student". (All numbers approximate, since I don't have your full dataset, but you can easily compute them by yourself.)
oops, confidence level -> confidence interval.
ReplyDeleteI'm curious as to what your exact definition of the term "cognitive threshold" is... To me, this phrase suggests some kind of strongly nonlinear effect -- cognitive ability exceeding the threshold allows understanding, which leads to much higher performance, while cognitive ability below the threshold precludes understanding, leading to much lower performance. There is no such nonlinear effect that is obvious to me in this graph -- it looks a whole lot like a standard, linear, reasonably strong correlation.
ReplyDeleteBut you seem to be saying that a purely linear model can imply a cognitive threshold... If so, what is the difference between the more quantitative fields such as mathematics and physics, which you say do show a threshold, and less quantitative fields, which you say do not? Just the value of the correlation coefficient, or the slope of the best-fit line? Or something that can better justify a binary classification of fields into those with and without thresholds?
1. The 600 number is approximate, just to quote a round figure
ReplyDelete2. Yes, we'd like to have a lot more data; note I said "modest evidence for a cognitive threshold".
3. "mastery" is of course a subjective definition. I would point out that even at the threshold I defined most of these students don't actually have mastery -- when they take the grad school exams (prelims or masters exam) about half will fail. Ask any professor who teaches grad QM or grad EM whether the incoming grad students actually have mastery of the undergrad curriculum. At Princeton or Caltech, maybe so, but not at a typical grad school.
4. Alex, I think there may actually be a nonlinearity. It would be more evident if the lower left corner was filled in -- i.e., if all the kids who took a solid physics/math class, did poorly, and switched majors, had somehow instead stayed in the program. Then I think you'd see a "bend" in the distribution. (The upper left corner remaining empty.)
I suspect there is a ceiling to what hard work can accomplish, if aptitude is too low. I see this myself when I teach physics 101 to non-majors. Many very hard working, very conscientious students (i.e., they are camped out in my office hours, ask for additional problems and material to study, get A's in all of their other, e.g., humanities or social science, courses, etc.) simply cannot work basic physics problems or master basic concepts, even after an entire quarter of effort. Demonstrating this fact (which is obvious to any experienced professor) in a systematic study is difficult because students self-select their courses and will not continue in a particular direction once they realize the material is just too hard for them.
Of course it is possible that a linear (Gaussian) model still works but that below SAT-M 600 the students with sufficient conscientiousness to obtain a 3.5 GPA are just becoming exponentially rare. In that case we'd see some filling in of the upper left space, say between SAT-M 500-600. As always, more statistics would help. But my feeling is that an additional SD of effort becomes less effective as the ability level drops. (Perhaps we need a multiplicative model?)
Wow, according to that plot, you admitted someone with a SAT_M of about 400 to your physics or math program, and they got better than a C average. That person must have been very motivated; 400 is borderline mentally retarded isn't it?
ReplyDeleteLiam: have a look at more of our data. You will see some amazing cases of people with low SAT scores who nevertheless excel in college. I think there are soc majors with GPA > 4.0 and SAT combined of 800 or so! No test tells the whole story...
ReplyDeletehttp://abyss.uoregon.edu/~js/sat/sat_combined_gpa_all.html
http://abyss.uoregon.edu/~js/sat/
From this post I see Steve is still totally out of touch with reality.
ReplyDelete1.
"One might reasonably associate "mastery""
NO! A reasonable association would be with cummulative objective exams like the GRE subject test in mathematics or physics.
2.
There is a third factor. That is the extent to which the student fits and is suited to formal education in general (as it is carried out in the US) and the UO in particular, to what extent he can engage, to what extent he identifies with the grade making process, to what extent he is enthusiastic, to what extent he can make use of his fellow students, etc.
3.
The "two factor model" of Steve's shows only his own inability to struggle out from under ideology. Here, the ideology of American formal education.
Steve plays a language game of "mastery", "cognitive", etc. unaware that he is playing a language game.
4.
My bona fides is an SAT-M=800.
But Steve should be pitied as his own ethnic group has never produced any philosophy. Why? It must be genetic. Steve is biologically incapable of thought.
ReplyDeleteClassic statistical problem.
ReplyDeleteTen random students from the same group take an exam and all ten of them fail.
What can be said about the probability for the 11th student from the same group to pass?
Common sense suggests that he's going to fail, maybe because there is a cognitive threshold that keeps the whole group from attaining mastery of the subject.
Try to plug these numbers into statistical formulas for binomial proportional confidence intervals (Clopper-Pearson or Agresti-Coull), and you will find that all you have is a 95% upper limit of approximately .31.
If you work in an environment where normal exam passing rate is around .50, and you want to institute a policy that keeps students of this group out of the university, you have to do a lot better than .31 to justify this policy.
One thing this graph shows unambiguously is that grades are a very bad measure of abaility, and therefore that the only two countries that use grades (the US and Canada) have an elite which is less intelligent than that of other countries.
ReplyDeleteOr to put it another way, US higher education does not make efficient use of the natural human resoucres of the US.
Or yet another way, the US is a despicable, evil country which deserves destruction.
I wonder if anon has finally come out of the closet.
ReplyDeleteOf course the motivation for this post is transparent.
ReplyDeleteLooking at the evidence Steve sees the abysmal correlation between ability and grades.
Being the smartest is part of Steve's identity.
He therefore must convince himself that hard work, though it is enough to make good grades in other fields, is not enough in his.
Like I said, it is transparent.
It is strange that someone as smart as Steve cannot see how transparent he is.
I get both the motivation and the skepticism about what the data here show. The main issue, I think, is that we don't observe the predicted failures of low-SAT students in your figure, probably because they go to other fields. Is the lack of low-SAT students unique to math and science? Do we see the exits, with the left tail moving into other fields? Some kind of comparison with other fields might be the best way to see this.
ReplyDeleteSorry, I guess your claim is that math/physics looks different, which may be in the other figs. Might be more obvious if you put them all together. Maybe use red for the population, blue for math/physics.
ReplyDeleteSo the people that are really bad at taking a relatively easy math test (the SAT's) don't get very good math GPA's, GPA's which are primarily calculated by how well students do at various math tests. This seems to border on being a tautology, people who are bad at taking a math tests will not be good at taking more math tests. if anything I'd say that the most interesting feature is how low the correlation is.
ReplyDeletethe low correlation is not a surprise to me.
ReplyDeleteIt was a surprise to Steve. But Steve fell prey to the "just-world phenomenon" as is his wont and that of almost all in his social position.
That is, the low correlation was a surprise to Steve. It challenged his world-view.
BUT he INSTANTLY came up with an explanation to save his world-view.
1.
It is a good thing that God given ability does not determine college "success", that the virtue hard-work can make up for Nature.
2.
Although hard-work is enough in other fields it is not enough in Steve's, and therefore, in a manner of speaking, Steve can continue blissfully thinking he is the world's smartest man.
There's more recent data for the SAT math percentiles for various groups.
ReplyDeletehttp://professionals.collegeboard.com/profdownload/SAT-Percentile-Ranks-by-Gender-Ethnicity-2009.pdf
The Asian scores seem to have gone up a bit relative to the other groups. A 750 on the math section is only at the 88th percentile level for Asians overall. Undoubtedly this percentile would be even lower if the category focused on East Asians alone.
"Undoubtedly this percentile would be even lower if the category focused on East Asians alone."
ReplyDeleteWRONG! Typical Chinese racist.
Subcontinentals outscore Chinese, Koreans, etc.
Are Saudis classed as Asians here?
I'm guessing that Indian Americans, if that's who you're referring to, have lower SAT math scores, but higher SAT verbal scores relative to East Asians. But even if they had slightly higher SAT math scores, I'm assuming that removing South East Asians and Pacific Islanders from the total would result in a net gain if one were to focus on East Asian scores alone.
ReplyDeleteBy the way, when did posting or speculating about nothing more than hard statistics qualify one as being a racist? I never speculated as to the reason why certain groups tend to score lower than others. :)
ReplyDeleteRegarding the history/english graph:
ReplyDeleteYet another reason to HATE America and its professors.
"There is a third factor."
ReplyDeleteThere's a fourth factor. Some students underachieve on SATs because they don't give a shit about them. Says nothing about their lack of natural ability.
It's not an error, but it must be acknowledged: the "average" motivation of a student at college is very low from my perspective.
ReplyDeleteOh, sure, people everywhere will *say* they're ambitious and motivated, and *think* that they are, but in fact few students spend more than 1 hour out of class for every hour in class, certainly *not* 3 hours.
And I'm saying that this needs to be taken into account, because I expect that anyone with a Sat-M over 700 could get a 3.5 GPA in a physics major at a vast majority of universities.