Sunday, February 26, 2012

Test preparation and SAT scores

How easy is it to raise SAT scores via test prep? The figures below come from The Effect of Admissions Test Preparation: Evidence from NELS:88, by D.C. Briggs.

Black dots are students who participated in test preparation (e.g., an expensive commercial course, or even a private tutor; see below) between taking the PSAT and taking the SAT. White dots are students who did not. Click for larger versions.

But who needs data like this when one can simply assert that it is obvious that test scores reflect SES or preparation, rather than actual ability? Even a casual investigation into this topic reveals that, at least on average, SAT scores are not easily improved, even through extensive effort. (Insert IQ or g score for SAT score if desired.)

Biggs: ... When researchers have estimated the effect of commercial test preparation programs on the SAT while taking the above factors into account, the effect of commercial test preparation has appeared relatively small. A comprehensive 1999 study by Don Powers and Don Rock published in the Journal of Educational Measurement estimated a coaching effect on the math section somewhere between 13 and 18 points, and an effect on the verbal section between 6 and 12 points. Powers and Rock concluded that the combined effect of coaching on the SAT I is between 21 and 34 points. Similarly, extensive metanalyses conducted by Betsy Jane Becker in 1990 and by Nan Laird in 1983 found that the typical effect of commercial preparatory courses on the SAT was in the range of 9-25 points on the verbal section, and 15-25 points on the math section.

One of the most remarkable aspects of this line of research has been the lack of impact it has had on the public consciousness. ...

... By far the largest effect sizes belong to the those preparation activities involving either a commercial course or private tutor [NEVERTHELESS THE SCORE CHANGES ARE NOT LARGE], and the effects differ for each section of the SAT. On average students with private tutors improve their math scores by 19 points more than those students without private tutors. The effect is less on the verbal section, where having a private tutor only improves scores on average by seven points. Taking a commercial course has a similarly large effect on math scores, improving them on average by 17 points, and has the largest effect on verbal scores, improving them on average by 13 points. With the exception of studying with a book, no other activity analyzed in this manner has an effect on test score changes that is statistically different from zero at a .05 significance level.

... Does test preparation help improve student performance on the SAT and ACT? For students that have taken the test before and would like to boost their scores, coaching seems to help, but by a rather small amount. After controlling for group differences, the average coaching boost on the math section of the SAT is 14 to 15 points. The boost is smaller on the verbal section of the test, just 6 to 8 points. The combined effect of coaching on the SAT for the NELS sample is about 20 points.


maxw3st said...

It seems fairly intuitive that the best "test prep" is to actually have an education prior to the tests. If prepping at the last minute were a viable alternative it would negate the need for K-10 education. It is that process of education that the US has lost over the last 30 years.

steve hsu said...

To be fair this data does not rule out the possibility that a yellow horde of really determined kids, driven by fearsome Tiger moms for most of their lives, couldn't systematically raise their scores somewhat. But it does show that it's not an easy task. It's also true that at some point training for the test becomes actual education as opposed to mere test prep.

Even kids with the resources to hire private tutors didn't change their scores very much, on average.

botti said...

***One of the most remarkable aspects of this line of research has been the lack of impact it has had on the public consciousness. ...***

Certainly judging by a number of the comments to the NY Times the other day on the affirmative action debate.

Anonymous_IV said...

Still education is not the same thing as IQ or g, right?

steve hsu said...

Very hard to disentangle if you are talking about sub-populations with very different cultures and intensities of educational experience.

Matthew Carnegie said...

Well, in theory the differences between education and test prep are really that in education, information is supposed to be long term retained and is generally applicable for them outside the test. 

Not sure that's often that's actually true in practice (for most people)...

Matthew Carnegie said...

There might be something I've glossed over in reading (or fundamentally misunderstood), but I'd guess people who take work hard to prepare for the SAT were probably working hard to prepare for the PSAT. They're probably not people who are all easygoing about the PSAT and then work hard on the SAT.

Is this being controlled for? It would seem otherwise this would shows that any *intensification* of test preperation through coaching between the PSAT and SAT is ineffective at building or maintaining high ability, particularly when controlled for other things with which it may be redundant, such as ethnic background (as a proxy for studiousness) and hours of homework. It doesn't really make so strong a comment necessarily about how a lazy person who hadn't studied hard prior to the PSAT would respond to coaching (although regardless, it does seem like the mainstream is pretty wrong about the prevelance of lazy people who then take coaching).

steve hsu said...

Look at the range of scores. It seems unlikely that this is a special population of people. I suspect very few people who score that low on PSAT were "intensely preparing". More plausible that there is just some probability p that a given kid will try some test prep in between taking the PSAT and SAT. There might be some selection bias in the direction you mention but I doubt it is very significant.

Note there is some systematic improvement for very low scorers. Perhaps for them a few tricks help a lot. If so, they didn't know the tricks when they took the PSAT.

dadanada said...

It looks like the variance of the coached group is less than the uncoached group.  That may just be an artifact of the graph though.  
Also it looks like the majority did a little better on the SAT than the PSAT.  Hard to pinpoint the cause of that - test differences or more familiarity with the testing process or what?
Obviously no one should expect to gain hundreds of points from coaching.  But Steve, if you have your way with your version of meritocracy, then kids will all be scrambling for an extra 10-20 points on their SAT scores.  It will be necessary to do so.  
At least until genetic engineering renders it all moot. When your BGI groups starts breeding 200+ IQs, do you think they will be allowed to displace everyone else in education and throughout society?  Hmm, are you laying the groundwork for that today - getting us all to accept the idea that more IQ means more deserving and nothing else should really matter?  

steve hsu said...

Exactly how PSAT score predicts SAT score is a bit tricky because kids are still learning a lot at that age. In particular, some kids are only just encountering/mastering some of the material that is on the tests, like algebra or geometry or how to read a graph.

In E. Asia most parents/kids are already scrambling for whatever edge they can get on entrance exams, etc. Not sure this is a good thing, but on the other hand people seem a bit better educated there on average. As long as it's not test-specific tricks they are working on, there is likely some actual learning/education going on. When I first visiting Seoul I was shocked to turn on the TV in my hotel room late at night and find someone working out calculus problems on one of the channels!

Edwin said...

I have posted this here before:you can hack the SAT.I  increased my score by 140 points without taking a  test prep class.Paradoxically,the section they regard as the hardest to hack-the verbal section was the one in which i improved by 100 points.

Yan Shen said...

 There's an extremely interesting theory that I've read on various blogs that more or less states that no matter how hard blacks and Hispanics prepare for an exam, they can never close the gap with whites, but that virtually all of the East Asian-white disparity on any kind of standardized exam can be attributed to East Asian conscientiousness. To say the least, I've always found that to be an extremely amusing and incoherent idea...

tractal said...

If you were less rabid this great paradox would make more sense. Exam specific test prep =/= life long differences in educational intensity.  The question that you keep harping on is really just whether there are significant differences in the white-asian educational experience/test prep. 

There are a few reasons why this situation is not quite analogous to the white-black gap. Most importantly, a lot more research has been done. A fairly stable 1 SD difference shows up in all of the tests, consistent with the 1 SD difference found in direct IQ tests. East Asian SAT performance cannot be explained by simple reference to IQ, especially on the SAT-V. Less gifted Asians are more likely to take the test, 'Asians' include groups who perform less well than whites, and many Asians are learning English as a second language, which is a disadvantage on the SAT-V. Nonetheless, Asians basically match whites on the SAT-V. It is hard to explain that in terms of G, whereas it is easy to explain the black-white gap in terms of G. G cannot account for the difference in SAT scores. 

Allright, so we have a problem. Do we have a hypothesis? The simplest one is that Asians tend to have more intense educational experiences, and may be more likely to invest in test preparation. Both of these are borne out by reasonable anecdotal information. The relevant social science tends to confirm our suspicion, but not univocally. Throw in differential Asian educational intensity and things start to make sense. 

You want to say that IQ tests showing East Asian advantages are really good measures, and that East Asian American over performance on the SAT is due to these advantages. But IQ differences can't account for the SAT performance. Nor can selective immigration hypotheses; EA American's perform like EAs on IQ tests. 

Lastly, consider how strange it would be if Asians were not benefiting from their more intense workload. Asians are far more likely than whites to take Calculus in high school, and to take AP classes. It seems crazy to suggest a more thorough math education doesn't help you on the SAT M. It is g loaded no doubt, but it also rewards mastery of math concepts. Someone taking Pre calculus in the 10th grade is far more likely to have a mastery of the relevant concepts than someone being introduced to Cartesian geometry, as professor Hsu noted. This is probably why the SAT-M is less G loaded and more susceptible to coaching than the SAT-V. 

Yan Shen said...

Asian Americans matching or slightly outperforming white Americans on the SAT-V at the high end of the scale, i.e. above 700 despite claims in the psychometric literature that East Asians lag slightly behind Europeans in verbal ability can easily be explained without recourse to Asian Americans having extensively boosted their SAT-V scores through preparation.

1. Selective immigration. There's no reason to assume that East Asian Americans are a perfectly representative sample of the East Asian population as a whole.
2. High Asian American SAT-V scorers are disproportionately Indian American, a group well known for being high in verbal ability, as evidenced by their recent domination of the spelling bee.

Yan Shen said...

 By the way Tractal, I read your comments on Half Sigma's post regarding the NYT article on Asian American over-representation/African American underrepresentation at Stuvyesant. It has already been pointed out that high IQ East Asians/Jews are more likely to apply to elite colleges relative to high IQ white Americans. This probably explains why there are more East Asians and Jews at elite colleges relative to what their IQ advantage over white Americans might predict. My guess is that an analogous situations exists with respect to elite math/science high schools.

That coupled with the fact that the Stuvyesant entrance exam loads heavily on mathematics, probably explains why 72.5% of Stuvyesant is now Asian American. From the NYT article...


Rudi took Kaplan’s 12-week program, which met on Saturdays at Fordham
University, at a cost of $750, the summer after seventh grade. (Students
take the exam in October of their eighth-grade year.) Her tutor, a
Stuyvesant graduate, persuaded her to make the school her first choice

Her mother, Annmarie Miller, a nursing assistant at a hospital in the
Bronx, recalled a cousin’s reaction when she mentioned Rudi’s pick: “You
have to be Chinese or Indian to get in there.” A co-worker, also black,
“said the exam is built to exclude blacks because it’s heavy on math,
and black people can’t do math,” Mrs. Miller said.

Rudi said she has never felt uncomfortable at Stuyvesant, but she has
felt puzzled. She has been the only black person in most of her classes,
and often goes hours without seeing another. The school’s attendance
sheets have names and pictures of the students, and she said teachers
were quick to learn who she is; there are few others like her, she said."

tractal said...

Not very convinced, but the Indian American point is interesting (I would have to see data... spellingbee=SAT-V  talent is not an argument at all). As for the correlation-causation in high school curriculum, I think its pretty obvious that there are reasons for the intensity of Asian coursework other than G. Saying that Asians take harder classes more just because they are smarter is implausible on many levels. 

At this point it seems like you are trying to deny the Asian parent expectation phenomena, and I just can't see why that is a necessary or reasonable conclusion. 

As for the selective immigration hypothesis, there is definitely reason to think EAAs are representative of the EA population. EAA's match the general psychometric trends of EAs. I don't know about the standard deviation thought, it might be higher. 

tractal said...

Yeah, maybe. But 72% is still far higher than G would predict. The cut off score isn't even that high, so it is not a matter of larger absolute differences at the tail. 

Yan Shen said...

 "My point is just that 72% Asian is probably not accomplishing the
meritocratic mission of getting the brightest kids into the classrooms
they need."

But this misses the point. If high IQ Asians are more likely to apply to
elite high schools relative to high IQ whites and both groups are
admitted at the same rate ceteris paribus, then you automatically have a
situation where there is a larger percentage of Asian Americans at
elite high schools than would be the case if the schools simply took the
highest IQ kids above some threshold in rank-order. No need for recourse to theories about Asian Americans grinding it out preparing for the entrance examination.

"I guess I would recommend making applicants document whether they had
received test preparation and then taking that into account, or finding a
more G-loaded test."

Well, I don't understand this g business too well, to be honest. But I'm not sure it works in your favor, given what the psychometric literature states.

"Only one study to date has examined East Asian–White difference on psychometric tests as a function of their g loadings; it confirmed the hypothesis for 15 cognitive tests
administered to two generations of Americans of Japanese, Chinese, and Europeanancestry. In this case, the more g-loaded the test, the greater the mean East Asian–White group difference favoring East Asians (Nagoshi, Johnson, DeFries, Wilson, & Vandenberg, 1984).

Perhaps you're simply invoking Sineruse's argument that the East Asian American inability to prep for a more g-loaded exam would decrease their scores relative to whites by more than what they would gain from the test being more g-loaded.

tractal said...

I'm open to some of it being self-selection, like I said. But self selection has a hard time explaining such a dramatic result. 

So it looks like the admissions process is susceptible to 1) prepping or 2) earlier educational experience. There is an obviously legitimate argument to be made that 2) should be a criteria, but that is a different question. My underlying thought is that gifted education is important for cognitively gifted kids, and you should try to get the cognitively gifted into appropriate classrooms. A 72% Asian result suggests that other factors are intervening, though it doesn't prove it. 

Yan Shen said...

 "When I first visited Seoul I was shocked to turn on the TV in my hotel
room late at night and find someone working out calculus problems on one
of the channels!"

Perhaps if you had flipped through a few more channels, you could've seen some epic Starcraft matches... :)

Yan Shen said...

Only time will tell whether or not the current cohort of East Asian Americans under-perform in math/engineering/computer science and the physical sciences relative to their test scores. Keep in mind that Jews dominated academically in America in the early 1900s, but my understanding is that they didn't become intellectually preeminent in the United States until around the time of World War 2 or slightly afterwards. In the meantime, I believe that Steve Hsu has cited evidence suggesting that the SAT has roughly the same predictive ability across different races.

tractal said...

Yeah, obviously we don't know and wouldn't want to generalize. But we have started to see the negative effects of Asian striving in Law and Medical bar passage rates. 

Yan Shen said...

 Are law and medicine both verbally loaded fields? Have I ever claimed that East Asians excelled at verbally loaded fields relative to white Americans?

tractal said...

The cool thing about verbal IQ is you can keep track of arguments and not get confused attacking the wrong claim. 

You are entirely missing the point of my last post. I wasn't saying anything about EA aptitude, I was just observing that in line with striver predictions, EA test scores and grades over predict performance on things like the USMLE and bar exams. So as a result of striving artificially inflating test scores and grades, some students get to the end of the process and are simply unable to pass the final hurdle, which is really tragic for them. The systems designed to ensure that this kind of thing doesn't happen are apparently susceptible to Asian striving. 

Yan Shen said...

 Or maybe the bar exam loads more on V and less on M relative to the LSAT, which has a logic puzzles section.

tractal said...

You are welcome to argue with the US medical board that they need a better test. 

Yan Shen said...

 "The cool thing about verbal IQ is you can keep track of arguments and not get confused attacking the wrong claim."

This coming from the guy who laughably tried to equate critical thinking entirely with verbal IQ.

tractal said...

If you had a higher verbal IQ you would know that I never claimed that. 

Yan Shen said...

 You kind of backtracked and did a 180 after it was pointed out to you how ridiculous your initial assertion was.

tractal said...

Nope, its archived if you really want to go see. I said "Of course we may want our citizens to be able to evaluate arguments (what is often called critical thinking) but that is just verbal IQ" 

Yan Shen said...

 "being able to spot fallacies is really important."

Here's a major fallacy. I'll simply insist that the SAT is significantly amenable to test prep and ignore every single Steve Hsu post citing studies that argue to the contrary.

tractal said...

I'm not ignoring it, though I do think it is surprisingly low. 

The fallacy here is 

Tractal thinks Asian SAT test scores may be subject to a striver effect

Therefore Tractal believes the SAT is subject to test preparation equal to the purported striver effect

This reasoning is fallacious, because it fails to consider the possibility that Tractal thinks Asians have generally different intensities of educational experience, which would produce a striver effect relative to G, but perhaps not an SAT specific striver effect. 

Yan Shen said...

"Appeals to authority, moralistic fallacies, correlation causation, concord fallacies, ad hominem, etc."

Interesting because IIRC you, along with quite a few of my detractors here at this blog, failed to understand what an ad hominem actually was. You accused me of engaging in an ad hominem attack against Sineruse when I pointed out that he was obsessed with East Asians. Of course, I pointed out to you then that it's not an ad hominem to criticize a person's character. It's only an ad homimem when you try to refute an argument by criticizing a person's character.

brown_slacker said...

"But who needs data like this when one can simply assert that it is obvious that test scores reflect SES or preparation, rather than actual ability?"

If it reflects preparation I don't see why that's inherently bad. Any good system of incentives ought to reward good behavior. A solid work ethic (test prep) and investing in one's child's education are examples of good behavior and ought to be rewarded. 

Also, there is enough capacity in undergrad institutes in the US to give every competent kid a solid education. This is borne out by statistics that you've yourself posted about how, till very recently (prior to finance increasing income variances and US's Gini coefficient), the incomes of people are predicted quite well by their SAT scores, irrespective of which college they attended. So where is the real issue? 

Yan Shen said...

 "So where is the real issue?"

White Americans built this country. Non-whites are overrunning this country. Yada yada. Anyone with two brain cells can see that all of this white nationalist/neo-Nazi rhetoric about East Asians boosting their SAT scores through test-prep is just a smokescreen for the real underlying motivation.

tractal said...

Your right, I don't understand the Ad hominem fallacy. Since, I don't lets roll with it: 

Yan Shen is a lonely, obsessed beta male Asian supremacist and ethnic chauvinist. Therefore, his arguments should be discounted by everyone. 

Yan Shen said...

More evidence that there are profound behavioral differences between Europeans and East Asians. Your comment might be considered clever, if it weren't so incredibly childish and juvenile.

brown_slacker said...

Surely Dr. Hsu would  refrain from getting into a slanging match with neo-Nazis. If this post is directed at liberal NYT commenters, I think Dr. Hsu should not be apologetic about preparation being different across different demographics (even if it has no demonstrable statistical effect in terms of inflating scores significcantly) ... chance favors the prepared mind and all of that.  

Yan Shen said...

 You know, reading that NYT article about Stuvyesant just pisses me off.

"Rudi said she has never felt uncomfortable at Stuyvesant, but she has
felt puzzled. She has been the only black person in most of her classes,
and often goes hours without seeing another. The school’s attendance
sheets have names and pictures of the students, and she said teachers
were quick to learn who she is; there are few others like her, she said."

Guess what, I went to a public high school in the suburbs of Houston that had very few Asian Americans. A lot of the Asian Americans gravitated towards a more elite high school within my school district. Often I was the only Asian American in most of my classes, or perhaps at most one of a couple. Did I ever think to myself, "Gee, how terrible it is that there are so few people in my classes who look like me!" No, I focused on what I needed to and took care of business as best as I could.

I mean Jesus fucking Christ, why is this country full of whiners and crybabies? I'm so sick and tired of listening to blacks and Hispanic whine about white privilege or whites whining about East Asians grinding it out preparing for exams. Why can't black, Hispanic and white Americans take a page out of the East Asian playbook and learn how to 吃 苦, or eat bitterness. Instead, all non-East Asian Americans in this country do is bitch and whine and moan.

Plenck said...

So the NYTimes and the people that comment there are all white nationalists and neo-Nazis?

Yan Shen said...

I've pointed out before that the most ironic/amusing thing is that when it comes to explaining East Asian American academic performance, liberals and white nationalists/neo-Nazi's invoke the exact same argument. However, the major difference is that liberals also insist that white Americans score higher than blacks and Hispanics because of more extensive access to test prep, while white nationalists/neo-Nazis adamantly insist that no amount of preparation can ever help blacks and Hispanics close the academic achievement gap with whites.

David Coughlin said...

Oh, the both of you ...


David Coughlin said...

I think that is a deep philosophical question that is getting more-and-more challenging to answer in the internet-as-encyclopedia-of-knowledge world.

Kevin Rose said...

I love how you make a fierce post, and then in the comments section you yourself provide the logic that totally undermines your own original point. You have done this a few times already. I guess you really do have an intellectual conscience after all ;) I admire that, especially since upon reflection, your sheer intellectual honesty always seems to win out over your ethnic advocacy. 

David Coughlin said...

Speaking of education and assessing outcomes, do you have a position on 'Everyday Math'?

dwbudd said...

I read the article, which was pretty typical of the sort of "isn't this terrible" rhetoric one would expect from a newspaper that thoroughly embarrassed itself by pushing an unqualified writer to bolster its "diversity" (i.e., The New York Times, and Jayson Blair).

Not sure WHY the 72% figure reported would be higher than expected based solely on the shaky "g" hypothesis.  It seems reasonable that there might be self-selection (that Yan Shen argues.)  It's also entirely possible that the relatively low percentage of white students at Stuy is the result of a different type of self-selection:  namely, that high-IQ white students may be economically more likely to opt for, say, The Dalton School or other private high schools, compared to Asians?

Stuyvesant is a public high school; following the "white flight" from New York City some years ago, I would be keen to see what percentage of middle-class white parents remain in Brooklyn or Queens, compared to Asian immigrants or second generation Asian Americans.  I would suspect that relatively speaking, whites left behind in New York are either very wealthy (living in TriBeCa or the Upper East Side) or lower-middle class/blue collar.  Conversely, I would suspect that there are still sufficient numbers of upper middle-class Asian families living in Flushing or other such places with bright children. 

The article mentioned it could not track those who were in the private school pipeline, but I strongly suspect that that reality has an impact on the numbers here.  I'd like to see the data, as it's just that - a suspicion.

gwern0 said...

"My new kung fu is unstoppable!"

RKU1 said...

This seems pretty plausible.  I think a related factor might be a cultural fear/hostility to public education among elite NYC American whites, which grew over the last few decades of heavy media coverage of dangerous and dreadful public schools.

For example, there was an interesting NYC article a week or two ago about how European elite families living in Manhattan typically send their children to the local public schools, while almost none of their American peers would even consider it.  Apparently the reason is that the Europeans come from a background in which they assume the local public schools will be fine, and so consider the ones in NYC with an open mind, while the local American elites start with very different preconceptions.  

It's possible that the Europeans make slightly too positive assumptions of the NYC schools, but I do suspect their perspective is much closer to the truth.  Meanwhile, the Americans might spend $40,000 per year per child tuition on private schools, and can barely make ends meet on their $500K annual salaries.

Add the need for two hours of subway commute to Stuyvesant, located what used to be a pretty dangerous neighborhood I think, and the shortage of elite white applicants to Stuyvesant isn't too surprising.

Ken Condon said...


Yan-you must purchase (and then wear of course) a set of finely hand stitched cowboy boots, a pair of boot cut Levis, a hand tooled and studded thick leather belt with a copper buckle with cow skull, a nice cotton plaid long sleeved shirt with bolo tie, and a Stetson. Side arm optional. Be sure to soak the Stetson in water for one hour and then wear it on your head until it dries in the sun to get the perfect fit.

Then mosey on in to the center of town tell everyone who scored lower than 750 on any of the SATS to clear out. “Theres a new sheriff in town-and I ain’t takin no fuckin shit from any of you morons!”

NRWO said...

WRT "A vocal subset of commenters there kept claiming that SAT is only a measure of SES or test prep."

Test-retest gains on the SAT are not g-loaded, suggesting that the gains are attributable to test specific strategies, which are not likely to have benefits beyond the test being studied (e.g., Coyle, 2006, Intelligence, Test-retest changes...). Such gains likely reflect practice effects in taking a test twice, as well as test-prep effects. More relevant: The CB conducted a meta-analysis of coaching effects, which concludes, in short, that coaching effects are either non-existent or very small in strength (from an effect size perspective).

A qualification: In theory, high-g people should be more likely to show test-retest gains, because they should be more likely to quickly ascertain (and learn) test-specific strategies that maximize performance. Some studies have reported a trend in support of this hypothesis.

Also, as has surely been noted: The SAT and ACT are de-facto intelligence tests, as documented in numerous reports (Frey & Detterman, 2004, Psych Science; Coyle & Pillow, 2008, Intelligence.). They correlate strongly with the latent construct that gives cognitive tests their predictive validity (i.e., g) and that, when removed, typically neutralizes predictive validity (but see, Coyle & Pillow, 2008).

Shan Barani said...

What is the IQ cut off to enter this school

botti said...

Hehe, well said Yan :)

dk1166 said...

If I can give my experience with test prep....


When I was in middle school they skipped me a few grades and put me
downstairs with the highschoolers. Everyone in the high school was required to
take a 2 day sat crash course. Which was structured as test, tutorial, Q&A,
retest. My scores were as follows on test 1 660V 330M. A 12-15 page packet and a Q&A
session later 640V 670M (on a different version of the test obviously). 340 points in a day on the math. Personal experience
says to me that the math at least is quite coachable. This was back in the early
90's so things may have changed. But I thought I would throw in my 2 cents

Carson Chow said...

Can you give the citation that shows that Asians have trouble passing the Bar or Medical Boards relative to their SAT scores?

tractal said...

Not easily. Law and Medical schools only keep track of LSAT and MCAT respectively. We could do some kind of demographic equating to determine SAT-->board exam predictive value, but that would take a bunch of data about self selection factors and who knows what else. It would be a minor research project. 

What seems clearer is that LSAT and MCAT scores overpredict Asian performance in the bar and USMLE. USMLE at least has received some actual study:

The LSAT-->Bar relationship is more treacherous for a few reasons. At any rate, Whites score slightly above Asians in LSAT mean but first attempt pass rate is 94.5%, Asians at about 86%. 

There are, however, important reasons not to over-interpret this. First, as is widely known, the Asian bell curve is really two bell curves, so Asian LSAT standard deviation is higher. In something like bar passage, you are really measuring how badly the bottom of the left half does, so we should expect Asian mean LSAT score to overpredict Asian first attempt passage rates. Moreover, there are a number of different bar exams. California's is notoriously hard, so if Asians are over-represented there this would also have an effect.

However, LSAC seems to have accounted for some of these issues: Per LSAT score and GPA, Asian individuals are significantly less likely to pass the bar than white candidates, which you can see at the bottom of page 52. Cautiously interpreted, this seems to match the medical case. 

Kevin Rose said...

Something interesting - self-discipline predicts performance on standardized tests (and a host of other academic metrics) better than IQ.

Mike Carter said...

Thanks so much for sharing this. Postings like these is why I have been helping my parents find an SAT tutor from the Bay Area to help my little brother. He has been struggling in school big time and if those charts are any indication of kids who are coached, then we are definitely making the right decision.

jaymesreyns said...

I just took my SATs a few months ago and I did terribly! I've never been the best test taker and I feel like I was totally unprepared for the timed portions. My parents are getting me a SAT tutor in the Bay Area, but I don't think that it's going to help with my test anxiety. Do you know of any tips to prepare yourself mentally for taking tests like these?

Clarissa Brooks said...

Scores of students who have taken SAT prep versus students who have not seem to be vastly spread around to my surprise. I would have thought students would score on average much higher with taking preparatory measures but I am sure it comes down to the student themselves. My son has to take his SAT test next year so I have been looking into SAT tutors in my area to help him prepare and I was hoping to see better finding then this. I think in either way a take towards preparation can only help improve a students score but also comes down to the quality of the course as well as student motivation and determination to do well.

aseuss said...

Many top graduates of elite medicals schools in Iran, India who have decent English language skills fail the USMLE on first attempt. A plausible reason could be that the passages are long--as I understand it, per 60-minute section (there are 7 in total) you have to read 50 page-length cases and answer the accompanying question. Time is therefore a major factor, and I would guess that most immigrants read and process English much more slowly than the average native-born, Caucasian candidate. The same is probably also true, to a somewhat lesser extent, for Asian American applicants for whom English may not have been their first language; in most Asian homes, a language other than English is spoken or at least English is not spoken in a manner that is conducive to mastery. (Intuitively, if you grow up in an environment where English is spoken incorrectly or not fluently, you will be at a disadvantage to your peers.) A somewhat weaker facility with English may manifest itself through performance on verbal-loaded (read: English-loaded) tests like the bar exams and the USMLE, which moreover impose a significant constraint in the form of time. Those with somewhat lesser English language ability probably invariably read more slowly than their peers, and therefore are doubly disadvantaged. Perhaps they don't finish each set of questions or are more rushed than those with a similar degree of knowledge and intelligence; taken in aggregate, this could account for lower passing rates.

I just don't think that high performance on standardized tests due to striving masks poorer doctoring or lawyering ability. If one can beat a standardized test like the SAT, MCAT or LSAT, one could presumably pass a content-loaded professional licensing exam that I'm guessing is less cognitively demanding. Again, the operative factor is probably English language skills. By the way, Asian group performance on the SAT-V exhibits a clear bimodal distribution: there are a large number of test-takers who perform very poorly, worse than any other group. These are the Asians who just don't speak English very well, and these will be the ones who probably have it harder when it comes to lengthy professional licensing exams.


I think getting a jump on the SAT TEST as soon as possible is something that everyone should consider especially if they want to start applying as soon as possible. I have been taking an online SAT prep course to help me prepare for the test, as it being online has been very convenient for me.

Sat Prep Test

prepify said... app provide great help to all the students Now the low income families kids can also get free learning lessons from Prepify ..!!

Stephanie Garner said...

Very good and important practice session for SAT best in all over the world. Registration process is very easy For all students. All free sat math prep ages of students can get benefit from this session don’t miss this chance I’ll also avail this Chance at any cost.

Sushant Rai said...

Useful and detailed information posted here. You can also get the details of gate institute in vijayawada here. Thanks for sharing.

Blog Archive