Non-linear Psychometric Thresholds for Physics and MathematicsThere is clearly something different about the physics and math GPA vs SAT distributions compared to all of the other majors we looked at (see figure 1 in the paper). In the other majors (history, sociology, etc.) it appears that hard work can compensate for low SAT score. But that is not the case in math and physics.
Stephen D.H. Hsu, James Schombert
We analyze 5 years of student records at the University of Oregon to estimate the probability of success (as defined by superior undergraduate record; sufficient for admission to graduate school) in Physics and Mathematics as a function of SAT-M score. We find evidence of a non-linear threshold: below SAT-M score of roughly 600, the probability of success is very low. Interestingly, no similar threshold exists in other majors, such as Sociology, History, English or Biology, whether on SAT combined, SAT-R or SAT-M. Our findings have significant implications for the demographic makeup of graduate populations in mathematically intensive subjects, given the current distribution of SAT-M scores.
One interesting question is whether the apparent cognitive threshold is a linear or non-linear effect. Our data suggests that the probability of doing well in any particular quarter of introductory physics may be linear with SAT-M, but the probability of having a high cumulative GPA in physics or math is very non-linear in SAT-M. See figure below: the red line is the upper bound at 95% confidence level on the probability of getting an A in a particular quarter of introductory physics, and the blue line is an upper bound on the probability of earning a cumulative GPA of at least 3.5 or so. The central values of these probabilities are much lower than the 95% confidence upper bounds: to have a 50% chance of GPA > 3.5 requires SAT-M in the top few percent of the population, judging just by the distribution of individuals in our sample. The confidence intervals are larger in the tails because the number of individuals is small.
Fig. 4.— Probability charts for SAT Math versus grades in Physics classes. For each SAT bin, the number of A-type grades (n) is listed along with the total number of grades per bin (N). The red line displays the 95% probability P(95%) and the blue symbols display an upper bound on the probability that a student will achieve 8 or more A’s out of 16 courses (typically necessary for an upper division GPA of 3.5.) Click image for better version.
Wouldn't you get the some nonlinearity every time you aggregate a number of alternative random variables? I.e. if your probability of tossing heads is 0.5 (red line). the probability of getting 8 or more heads out of 16 tosses is 0.5 (blue line). If your probability of tossing heads is 0.33 (red line), the probability of getting 8 or more heads out of 16 tosses is approximately (binomial distribution) 0.12 (blue line). In other words, the blue line MUST be nonlinear assuming that the probability of getting an A is linearly dependent on the SAT score and that all courses are independent random trials (that seems reasonable).
ReplyDeleteYes, that is our explanation of the effect (see figures in the paper for actual GPA vs SAT distributions). But note that in other subjects the probability of a cumulative GPA > 3.5 or even 4.0 is not so small even at low SAT score. I suspect that hard work can compensate more readily in those areas, allowing some students to have a higher single-term A probability (red curve) even when their SAT score is low.
ReplyDeleteAre you sure you have the right paper linked? It looks like a very early draft.
ReplyDeleteAnyway, did you do a logistic regression on the data?
cc
Oops. Bad link -- now fixed, I think.
ReplyDeleteDoes this map to IQ? Can you say a certain IQ is required to do well in physics or is it specific to math talent (which while very g loaded) is not IQ per se. Also, is there any limit to the power of hardwork outside physics? Could someone with an IQ below 80 do well in sociology with enough hard worK?
ReplyDeleteSAT-M is correlated with IQ, of course, but it's not the same thing. The M-V split is to me an obvious limitation to the g-factor construct. I know plenty of people with very high verbal IQs that are unable to understand very fundamental mathematical concepts. The threshold we found in physics/math is about 85-90th percentile in the general population -- if you are below that it appears no amount of hard work can compensate. For the other fields you can see results vs SAT, SAT-M and SAT-V in our earlier paper and the linked graphs. People with very low combined SAT -- like 800, which might be around 90 IQ -- can earn a 3.5 or 4.0 average in Sociology.
ReplyDeleteSounds like very interesting research you're doing; it just feels a little circular to use scores on a math test (SATM) to predict how people will do on other math tests (physics class). Now if you found that a certain SATV was needed for good marks in physics or a certain SATM was needed for certain marks in English, that would have more theoretical significance than making the obvious point that folks who don't understand math fail math classes.
ReplyDeleteIn his book, "Straight Talk About Mental Tests", The Free Press, A Division of the Macmillan Publishing Co., Inc., New York, 1981, pg. 12, Dr. Arthur Jensen cites the following four IQ thresholds:
(1) An IQ of 50 or below. This is the threshold below which most adults cannot cope outside of an institution. They can typically be taught to read at a 3rd or 4th grade level. However, they cannot normally function in the customary classroom setting, and they require special training programs.
(2) An IQ between 50 and 75. At this level of intelligence, they generally cannot complete elementary school. Most adults will need smarter help in coping with the world.
(3) An IQ between 75 and 105. Children in this IQ range are not generally able to complete a college prep course in high school.
(4) An IQ between 105 and 115. May graduate from college but generally, not with grades that would qualify them for graduate school.
(5) An IQ above 115. No restrictions.
The point is that SAT-M vs Physics/Math performance is qualitatively different from, say, SAT-V and English or History. Why is that?
ReplyDeleteAlso, it's useful to know where the SAT-M threshold for, e.g., becoming an electrical engineer is.
I see your points. As for why the relationship is qualitatively different, my guess is that in non-math courses, there's more subjectivity so it's easier for a professor to give high marks to students who work hard. Also it's possible to memorize or BS your way through non-math courses. In physics by contrast, I assume you either grasp the concept or you don't & certain concepts may be beyond most people no matter how hard they try.
ReplyDeleteIt would be interesting to apply your research to law school. Half Sigma claims you need an IQ of 140 to get good grades at Harvard aw school. I think he's wildly wrong, especially in light of the research you present with respect to English. On the other hand, law is more logic based & conceptual than other non-math domains.
Steve, independently we have come on subjects as "Psychometric thresholds for physics and mathematics", "Cognitive decline of the university population", "Decline of the humanities", "Higher education and human capital" to identical and similar conclusions. Therefore you and the readers of your blog on information processing should be interested in the following links:
ReplyDeletehttp://www.v-weiss.de/majgenes-full.html
http://mpra.ub.uni-muenchen.de/14600/
http://www.v-weiss.de/chaos.html
Volkmar
I don't think you need cognitive thresholds to explain the data. A simpler explanation is that mathematics and physics are cumulative, while other disciplines are not to anywhere near the same degree.
ReplyDeleteIn other words, to succeed in mathematics you must take four or five courses, in sequence, and a failure to do well in one of those sequential courses essentially dooms you in subsequent courses in the sequence. You are very unlikely to do well in second semester calculus if you got a B- in first semester calculus.
Any time that you have a linear relationship between performance and innate academic ability, and a sequential series of courses, the result in inherently non-linear. Probability of a good result overall depends in a four sequence course set on the fourth power of success in any one course.
The humanities and social sciences are not nearly so sequential. Success in medieval history is not necessarily essential to success in early modern European history. There are benefits, of course, from fruitful cross-germination, but they are at the margins and standard references can compensate for missing something the first time around in another major course.
Since students with high SAT-M have often completed many of the pre-requisites for higher division courses in math and physics (for example, many have completed a couple of semesters of calculus prior to entering undergraduate studies with a math or physics major), the sequencing issues are absent for them (or more accurately, the sequencing issues have disappeared in the pool of people who washed out in high school or chose not to be math or physics majors).
If one is attached to cognitive thresholds, one could alternatively describe them as fundamentally a product of sequential study requirements.
I've heard reported in the popular press studies that show pre-collegiate math success more closely linked to verbal standardized test scores than mathematics standardized test scores (a plausible result in my view as someone who got a BA in mathematics). SAT-M, alone doesn't map as well to IQ as SAT-Combined does, IIRC.
ReplyDeleteI am also inclined to think that one is more inclined to see cognitive thresholds on the low end, than on the high end. For example, literacy is pre-requisite to any other academic pursuit and some developmentally disabled people will never be able to reach functional literacy.
I look at data on LSAT scores, undergraduate GPAs and bar passage rates of law schools here (using publicly available information, rather than data mining). A law school "needs to be in the vicinity of a 2.8 undergraduate GPA and 149 LSAT score (a score which is at roughly the 40th percentile for LSAT test takers) for something in the vicinity of 75% of it students" in order to receive an eventual 75% passage rate (i.e. percentage of students who ultimately pass, even if not necessarily on a first try) on the bar exam. This suggests that one needs roughly an undergraduate 2.8 GPA and 149 LSAT score, with some room for compensation between the two related inputs, to pass the bar exam, which is roughly the 40th percentile of law school applicants.
ReplyDeleteNote also that differences in bar exam passage rates between California (low), and other states, is mostly a product of the fact that California allows unaccredited law school graduates to take the bar exam, while other states do not.
Math and physics grades in separate courses are less independent of each other than most disciplines.
ReplyDeleteIt is also worth noting that a few researchers (I don't have citations in hand), have suggested that the sequential course matter (both within courses and from course to course) is a factor in the proportion of women in physics and math and engineering.
ReplyDeleteThe thesis goes (and I am not endorsing or confirming it), that mathematical/spatial ability is impacted by monthly variation in estrogen levels, and that being at a low point in a few critical days each term confounds performance in the rest of the term. I mention it, because it suggests that a pedagogical strategy (repetition during the course of a term of key sequential points in non-four week cycles) that in theory would dramatically alter persistence levels and that could be tested empirically. The strategy might also reduce the sequential course attrition rate.
This blog on the Jump Math curriculum is pretty interesting: http://opinionator.blogs.nytimes.com/2011/04/18/a-better-way-to-teach-math/It's on the elementary curriculum so it's not directly applicable but it does suggest that indeed the problem with math and science achievement is in fact that the learning is cumulative and in fact that almost everyone can learn the material if it's presented correctly.
ReplyDeleteCan almost anyone learn multivariable calculus if it is presented correctly? I wish someone would show me how to present it properly :-)
ReplyDelete