Tuesday, December 07, 2010

Shanghai PISA scores



The Shanghai math (+1 SD) and science (+.75 SD) scores are almost a full SD above the OECD average of 500 (SD = 100). The top 10 percent of Shanghai math students are all above the 99th percentile for the US. See earlier post for links to Rindermann's work relating school achievement tests like TIMSS and PISA to national IQ estimates, and see here for earlier SD estimates using 2006 PISA data. (Finland has an anomalously low SD in the earlier data. A quick look at the 2009 data shows the following math SDs: Finland 82, USA 91, Korea 89, Japan 94, Germany 98, Shanghai 103, Singapore 104.)

Although Shanghai and Beijing are the richest cities in China, incomes are still quite low compared to the US. Average income in Shanghai is about $10k USD per annum, even PPP adjusted this is about $20k. People live very modestly by the standards of developed countries.

As noted in the comments, there are other places in China that score *higher* than Shanghai on college entrance exams or in math and science competitions. So while Shanghai is probably above the average in China, it isn't as exceptional as is perhaps implied in the Times article.

Taiwan has been moving to an American-style, less test-centric, educational system in the last decade. Educators and government officials (according to local media reports in the last 12 hours) are very concerned about the "low scores" achieved in the most recent PISA :-)

To see how individual states or ethnicities in the US score on PISA, see here and here.

NYTimes: ... PISA scores are on a scale, with 500 as the average. Two-thirds of students in participating countries score between 400 and 600. On the math test last year, students in Shanghai scored 600, in Singapore 562, in Germany 513, and in the United States 487.

In reading, Shanghai students scored 556, ahead of second-place Korea with 539. The United States scored 500 and came in 17th, putting it on par with students in the Netherlands, Belgium, Norway, Germany, France, the United Kingdom and several other countries.

In science, Shanghai students scored 575. In second place was Finland, where the average score was 554. The United States scored 502 — in 23rd place — with a performance indistinguishable from Poland, Ireland, Norway, France and several other countries.

The testing in Shanghai was carried out by an international contractor, working with Chinese authorities, and overseen by the Australian Council for Educational Research, a nonprofit testing group, said Andreas Schleicher, who directs the Organization for Economic Cooperation and Development’s international educational testing program.

Mark Schneider, a commissioner of the Department of Education’s research arm in the George W. Bush administration, who returned from an educational research visit to China on Friday, said he had been skeptical about some PISA results in the past. But Mr. Schneider said he considered the accuracy of these results to be unassailable.

“The technical side of this was well regulated, the sampling was O.K., and there was no evidence of cheating,” he said.

Mr. Schneider, however, noted some factors that may have influenced the outcome.

For one thing, Shanghai is a huge migration hub within China. Students are supposed to return to their home provinces to attend high school, but the Shanghai authorities could increase scores by allowing stellar students to stay in the city, he said. And Shanghai students apparently were told the test was important for China’s image and thus were more motivated to do well, he said.

“Can you imagine the reaction if we told the students of Chicago that the PISA was an important international test and that America’s reputation depended on them performing well?” Mr. Schneider said. “That said, China is taking education very seriously. The work ethic is amazingly strong.”

... Ever since his organization received the Shanghai test scores last year, Mr. Schleicher said, international testing experts have investigated them to vouchsafe for their accuracy, expecting that they would produce astonishment in many Western countries.

“This is the first time that we have internationally comparable data on learning outcomes in China,” Mr. Schleicher said. “While that’s important, for me the real significance of these results is that they refute the commonly held hypothesis that China just produces rote learning.”

“Large fractions of these students demonstrate their ability to extrapolate from what they know and apply their knowledge very creatively in novel situations,” he said.

10 comments:

  1. Yan Shen7:28 AM

    "Average income in Shanghai is about $10k USD per annum, even PPP adjusted this is about $20k. People live very modestly by the standards of developed countries."

    Yes, this is lower than the average incomes of certain groups in the US. ;) But don't point that out to the politically correct thought police though.

    "For one thing, Shanghai is a huge migration hub within China. Students are supposed to return to their home provinces to attend high school, but the Shanghai authorities could increase scores by allowing stellar students to stay in the city, he said."

    Okay, it's not unreasonable to assume that the population in Shanghai is self-selected IQ wise relative to the population of China as a whole, especially the actual testing population in the schools themselves.

    "And Shanghai students apparently were told the test was important for China’s image and thus were more motivated to do well, he said."

    Ah, there's the politically correct pseudo-logic you'd expect from a publication like the NYT! How much was the Shanghai PISA distribution shifted rightwards because the kids were merely told that the test was highly important?

    “Can you imagine the reaction if we told the students of Chicago that the PISA was an important international test and that America’s reputation depended on them performing well?”

    Let me guess, because of the stereotype of America persistently being a relatively poor performer on international tests like TIMSS and PISA, stereotype threat would destroy the psychological well-being of the American test takers in Chicago, leading them to be become so mentally distressed and disoriented that they would end up scoring far lower than they otherwise could have. ;)

    “This is the first time that we have internationally comparable data on learning outcomes in China,” Mr. Schleicher said. “While that’s important, for me the real significance of these results is that they refute the commonly held hypothesis that China just produces rote learning.”

    If true, how will certain parts of the blogosphere and other assorted critics and skeptics go on living?

    "Ever since his organization received the Shanghai test scores last year, Mr. Schleicher said, international testing experts have investigated them to vouchsafe for their accuracy, expecting that they would produce astonishment in many Western countries. "

    I suspect not so much to those familiar with HBD.

    ReplyDelete
  2. I'd like to see how the results of tests like PISA are related to IQ differences, and to what extent they are due to differences in school systems. In particular, I find it astonishing that the Finland system, which is relatively easy-going, produces very similar results as the East Asian systems that have a reputation for extreme competitiveness and cramming.

    Heiner Rinderman published a paper which showed that the data from international student achievement studies are highly correlated with Richard Lynn's international IQ data, but the latter data are of pretty crappy quality, so I'd like to see this studied more. They should give a sample of students both the PISA test and, say, Wechsler's test. This way they could also perform a factor analysis of the PISA subtests to obtain their factor loadings -- I suspect that the "Mathematical Literacy" test is as much a test of verbal as of math skills.

    ReplyDelete
  3. Hey Steve,

    Impressive, but let me play the skeptic. Why compare elite cities of the world's most populous country against the averages of entire countries, most of them far more diverse than Shanghai? What are the PISA scores for China as a nation? Are there SD figures that could support or argue against the purported narrower IQ SD figure for NE Asians?

    Singapore is a city-state, but why break out Shanghai, Macao or even Hong Kong at this point? Do any other countries also break out and report scores from their top cities or elite enclaves? Is there a PISA-SAT conversion table? It would be interesting to know where school districts like Cupertino, CA or even particular schools like Thomas Jefferson in VA would fall on the PISA scale.

    How g-loaded are the PISA exams? Wiki mentions the PISA tests for a broad range of specific concrete skills with some abstractions to measure educational systems effectiveness in creating an employable workforce. Is it highly preppable? What US exam would it be most like? Iowa Basic Skills, old SAT, new SAT, Achievement Tests?

    Are these scores for Shanghai consistent with the previous PISA or other standardized exam results from China? I've sat exams in Asia with students from China and the cheating was rampant. This was a stated concern, but what specific steps were taken to try to address this? Can a general Chinese IQ be extrapolated from these scores factoring in selective filtering of Shanghai and the nature of the PISA exams? Is this consistent with other sources and data points?

    Fugu

    ReplyDelete
  4. Tycny8:27 AM

    Might you kindly give examples of different math, reading and science items or type of items measured on this assessment?

    Thank you
    Rosanne Alberts, Phd
    Testing young children
    new york, new york
    New York City

    ReplyDelete
  5. Moises5:13 AM

    You cand find a map of the of PISA report results for maths and reading student performances by country at
    reading: https://www.targetmap.com/viewer.aspx?reportId=3355
    maths: https://www.targetmap.com/viewer.aspx?reportId=3356
    Hope you find them useful

    ReplyDelete
  6. Yan Shen9:27 AM

    Here's some more interesting information regarding the potential quality of human capital in China.

    http://blogs.ft.com/beyond-brics/2010/12/07/why-are-chinese-schoolkids-so-good/

    "There are other unanswered questions. Is Shanghai the exception or the rule in Chinese school standards? In some countries, major cities underperform the national average, but that seems less likely in China, given the coast-interior disparities. However, the OECD did look at some rural areas, and found they matched Shanghai’s quality."

    It appears that the OECD has data on other areas which they didn't release in the official 2009 PISA report.

    ReplyDelete
  7. George W Shen8:58 AM

    Thanks for the pointer, esmith. Interesting map. But i am not sure what it represents. Does it represent the original place where people are from or the place where people get their education or the place where they work? Based on my observation, majority students went to college in another cities. It's more true in top univerisities where every class only has 1 or 2 students from each province. If i am from JiangSu and went to Shanghai for higher education, am i counted in the Shanghai or JiangSu bucket? It is probably even more true nowadays with migrant workers whose childern would go to another city/province for education. Another thing is this map favors northern part of China (with the exception of GuangDong). JiangSu and ZheJiang are ranked surprisingly low. Even Shanxi is higher than JiangSu.....that can't be true. No offense o the people from Shanxi. :0) FuJian is low as well. The whole each coast area where people are supposed to be more highly educated than inland isn't reflected in this map.

    ReplyDelete
  8. George W Shen9:24 AM

    esmith, regarding the map, I have more doubts.....to say average education in Inner Mongolia is better than ZheJiang, it's almost like to say average education in Iowa is better than Massachusetts.

    ReplyDelete
  9. Yan Shen8:43 AM

    You might want to read this article I linked to.

    http://blogs.ft.com/beyond-brics/2010/12/07/why-are-chinese-schoolkids-so-good/

    "There are other unanswered questions. Is Shanghai the exception or the rule in Chinese school standards? In some countries, major cities underperform the national average, but that seems less likely in China, given the coast-interior disparities. However, the OECD did look at some rural areas, and found they matched Shanghai’s quality."

    Also read this post carefully. The first sentence, if true, is rather telling.

    "As noted in the comments, there are other places in China that score *higher* than Shanghai on college entrance exams or in math and science competitions. So while Shanghai is probably above the average in China, it isn't as exceptional as is perhaps implied in the Times article."

    ReplyDelete
  10. whoah this weblog is fantastic i really like studying your articles. Keep up the great work! You recognize, a lot of persons are hunting round for this information, you could help them greatly.

    ReplyDelete