SAT score distributions in Michigan

The state of Michigan required all public HS seniors to take the SAT last year (~91k out of ~107k total seniors in the state). This generated an unusually representative score sample. Full report. 

I'm aware of this stuff because my kids attend a public HS here.

To the uninformed, the results are shocking in a number of ways. Look specifically at the top band with scores in the 1400-1600 range. These are kids who have a chance at elite university admission, based on academic merit. For calibration, the University of Michigan median SAT score is above 1400, and at top Ivies it is around 1500.

Some remarks:

1. In the top band there are many more males than females.

2. The Asian kids are hitting the ceiling on this test.

3. There are very few students from under-represented groups who score in the top band. 

4. By looking at the math score distribution (see full report) one can estimate how many students in each group are well-prepared enough to complete a rigorous STEM major -- e.g., pass calculus-based physics.

Previously I have estimated that PRC is outproducing the US in top STEM talent by a factor as large as 10x. In a decade or two the size of their highly skilled STEM workforce (e.g., top engineers, AI researchers, biotech scientists, ...) could be 10x as large as that of the US and comparable to the rest of the world, ex-China.

This is easy to understand: their base population is about 4x larger and their K12 performance on international tests like PISA is similar to what is found in the table above for the Asian category. The fraction of PRC kids who perform in the top band is probably at least several times larger than the overall US fraction. (Asian vs White in the table above is about 6x, or 7x on the math portion.) Also, the fraction of college students who major in STEM is much larger in PRC than in the US.

This table was produced by German professor Gunnar Heinsohn, who analyzes geopolitics and human capital.

Note, I will censor racist comments.

Recent Papers on Socio-Economic Status and Student Achievement: Marks and O'Connell

I received the message below from Michael O'Connell, University College Dublin, and Gary Marks, University of Melbourne. 

See also this recent post: Social and Educational Mobility: Denmark vs USA (James Heckman), and links therein. 

Dear Scholar, 

 

There is a widely-held perception that many of life’s key outcomes are fundamentally driven by people’s socio-economic status (SES). More specifically, there is a view that children’s educational attainment is largely a by-product of their familial SES. As a consequence of this pervasive paradigm, much of the energy in seeking to ameliorate or resolve poor educational attainment is based around trying to use SES as a social lever. 

However, in the six papers listed below, published between 2019-2022, evidence has been gathered demonstrating that SES is only very modestly correlated with educational attainment. Furthermore, once a child’s cognitive ability is taken into account, even the modest link between SES and attainment diminishes to slight influence. This is true of datasets drawn from international groups of young people, as well as those from the US, UK, or Ireland. Future attempts to aid and study young people experiencing difficulty with educational attainment should be built on an awareness of the limited role of SES. 

Gary N Marks   Michael O’Connell 

1. O’Connell, M. and Marks, G.N. (2022) 

Cognitive ability and conscientiousness are more important than SES for educational attainment: An analysis of the UK Millennium Cohort Study

Personality and Individual Differences, 188 

https://doi.org/10.1016/j.paid.2021.111471 

Highlights Antecedents of educational attainment of great interest Dominant paradigm focuses on SES of children. Cognitive ability and conscientiousness have stronger record in research findings. Using new UK MCS longitudinal survey data, GCSE state exam performance assessed Cognitive ability and conscientiousness explained far more than SES measures 

 

2. Marks, G. N. (2021) 

Is the relationship between socioeconomic status (SES) and student achievement causal? Considering student and parent abilities

Educational Research and Evaluation, 10.1080/13803611.2021.1968442: 1-24. 

Abstract Most studies on the relationship between students’ socioeconomic status (SES) and student achievement assume that its effects are sizable and causal. A large variety of theoretical explanations have been proposed. However, the SES–achievement association may reflect, to some extent, the inter-relationships of parents’ abilities, SES, children’s abilities, and student achievement. The purpose of this study is to quantify the role of SES vis-à-vis child and parents’ abilities, and prior achievement. Analyses of a covariance matrix that includes supplementary correlations for fathers and mothers’ abilities derived from the literature indicate that more than half of the SES–achievement association can be accounted for by parents’ abilities. SES coefficients decline further with the addition of child’s abilities. With the addition of prior achievement, the SES coefficients are trivial implying that SES has little or no contemporaneous effects. These findings are not compatible with standard theoretical explanations for SES inequalities in achievement. 

 

3. Marks, G. N. and O’Connell, M. (2021) 

No Evidence for Cumulating Socioeconomic Advantage. Ability Explains Increasing SES Effects with Age on Children’s Domain Test scores 

Intelligence, 88    

https://doi.org/10.1016/j.intell.2021.101582 

Highlights Data analysed for five domains for children of the NLSY79 mothers study. SES effects increase for only some domains and not substantially. No increase in SES effects when considering mother's or children's prior ability. Effects of child's prior ability on test scores increase substantially with age. SES effects are small net of mother's ability. 

 

4. Marks, G. N. and O'Connell, M. (2021) 

Inadequacies in the SES–Achievement model: Evidence from PISA and other studies 

Review of Education, 9(3): e3293. 

https://doi.org/10.1002/rev3.3293 

Abstract Students’ socioeconomic status (SES) is central to much research and policy deliberation on educational inequalities. However, the SES model is under severe stress for several reasons. SES is an ill-defined concept, unlike parental education or family income. SES measures are frequently based on proxy reports from students; these are generally unreliable, sometimes endogenous to student achievement, only low to moderately intercorrelated, and exhibit low comparability across countries and over time. There are many explanations for SES inequalities in education, none of which achieves consensus among research and policy communities. SES has only moderate effects on student achievement, and its effects are especially weak when considering prior achievement, an important and relevant predictor. SES effects are substantially reduced when considering parent ability, which is causally prior to family SES. The alternative cognitive ability/genetic transmission model has far greater explanatory power; it provides logical and compelling explanations for a wide range of empirical findings from student achievement studies. The inadequacies of the SES model are hindering knowledge accumulation about student performance and the development of successful policies. 

 

5. O'Connell, M. and Marks, G. N. (2021) 

Are the effects of intelligence on student achievement and well-being largely functions of family income and social class? Evidence from a longitudinal study of Irish adolescents

Intelligence, 84: 101511. 10.1016/j.intell.2020.101511 

Highlights Power of cognitive ability and social class contrasted. Large representative sample from longitudinal study, waves 1–3, of 6216 children Outcomes were attainments, difficulties and relationships. Cognitive ability explained large amounts of variance. Social background only minor effects 

 

6. O'Connell, M. (2019) 

Is the impact of SES on educational performance overestimated? Evidence from the PISA survey 

Intelligence, 75: 41-47 

https://doi.org/10.1016/j.intell.2019.04.005 

Highlights Policy-makers overly attribute differences in educational performance to SES. PISA survey used to assess roles of parental education and household income. Combining them concealed differences in outcomes between rich and poor countries. Household income important in poor countries, parental education in rich countries.

Social and Educational Mobility: Denmark vs USA (James Heckman)

Despite generous social programs such as free pre-K education, free college, and massive transfer payments, Denmark is similar to the US in key measures of inequality, such as educational outcomes and cognitive test scores. 

While transfer payments can equalize, to some degree, disposable income, they do not seem to be able to compensate for large family effects on individual differences in development. 

These observations raise the following questions: 

1. What is the best case scenario for the US if all progressive government programs are implemented with respect to child development, free high quality K12 education, free college, etc.?

2. What is the causal mechanism for stubborn inequality of outcomes, transmitted from parent to child (i.e., within families)? 

Re #2: Heckman and collaborators focus on environmental factors, but do not (as far as I can tell) discuss genetic transmission. We already know that polygenic scores are correlated to the education and income levels of parents, and (from adoption studies) that children tend to resemble their biological parents much more strongly than their adoptive parents. These results suggest that genetic transmission of inequality may dominate environmental transmission.

  

See 

Genetic correlation of social outcomes between relatives (Fisher 1918) tested using lineage of 400k English individuals 

Game Over: Genomic Prediction of Social Mobility

The Contribution of Cognitive and Noncognitive Skills to Intergenerational Social Mobility (McGue et al. 2020)

Note: Denmark is very homogenous in ancestry, and the data presented in these studies (e.g., polygenic scores and social mobility) are also drawn from European-ancestry cohorts. The focus here is not on ethnicity or group differences between ancestry groups. The focus is on social and educational mobility within European-ancestry populations, with or without generous government programs supporting free college education, daycare, pre-K, etc.

Lessons for Americans from Denmark about inequality and social mobility 

James Heckman and Rasmus Landersø 

Abstract Many progressive American policy analysts point to Denmark as a model welfare state with low levels of income inequality and high levels of income mobility across generations. It has in place many social policies now advocated for adoption in the U.S. Despite generous Danish social policies, family influence on important child outcomes in Denmark is about as strong as it is in the United States. More advantaged families are better able to access, utilize, and influence universally available programs. Purposive sorting by levels of family advantage create neighborhood effects. Powerful forces not easily mitigated by Danish-style welfare state programs operate in both countries.

Also discussed in this episode of EconTalk podcast. Russ does not ask the obvious question about disentangling family environment from genetic transmission of inequality.

 

The figure below appears in Game Over: Genomic Prediction of Social Mobility. It shows SNP-based polygenic score and life outcome (socioeconomic index, on vertical axis) in four longitudinal cohorts, one from New Zealand (Dunedin) and three from the US. Each cohort (varying somewhat in size) has thousands of individuals, ~20k in total (all of European ancestry). The points displayed are averages over bins containing 10-50 individuals. For each cohort, the individuals have been grouped by childhood (family) social economic status. Social mobility can be predicted from polygenic score. Note that higher SES families tend to have higher polygenic scores on average -- which is what one might expect from a society that is at least somewhat meritocratic. The cohorts have not been used in training -- this is true out-of-sample validation. Furthermore, the four cohorts represent different geographic regions (even, different continents) and individuals born in different decades.

The figure below appears in More on SES and IQ.

Where is the evidence for environmental effects described above in Heckman's abstract: "More advantaged families are better able to access, utilize, and influence universally available programs. Purposive sorting by levels of family advantage create neighborhood effects"? Do parents not seek these advantages for their adopted children as well as for their biological children? Or is there an entirely different causal mechanism based on shared DNA?

 

The figure below appears in Low SES does not decrease heritability of cognitive ability (N=300k).

 

An Inconvenient Minority: The Attack on Asian American Excellence and the Fight for Meritocracy (Kenny Xu)

Kenny Xu is a brave young man. His new book An Inconvenient Minority: The Attack on Asian American Excellence and the Fight for Meritocracy expertly documents a number of unpleasant facts about American society that most major media outlets, education leaders, and social justice advocates have been obfuscating or outright suppressing for decades.

1. Asian Americans (not foreign students from Asia, but individuals of Asian heritage who are US citizens or permanent residents) have been discriminated against in admission to elite institutions of higher education for over 30 years. 

To put it bluntly, Asian Americans must, on average, outperform all other groups in order to have an equal chance of admission to universities like Harvard or Yale. If one were to replace Asian Americans with Jews in the previous sentence, it would describe the situation in the early 20th century. Looking back, we are rightfully ashamed and outraged at the conduct of elite universities during this period. Future Americans, and observers all over the world, will eventually have the same reaction to how Asian Americans are treated today by these same institutions.

2. Asian American success, e.g., as measured using metrics such as income, wealth, or education, is problematic for simplistic narratives that emphasize race and "white supremacy" over a more realistic and multifaceted analysis of American society.

3. Efforts to guarantee equal outcomes, as opposed to equal opportunities, are anti-meritocratic and corrosive to social cohesion, undermine basic notions of fairness, and handicap the United States in scientific and technological competition with other nations.

The Table of Contents, reproduced below, gives an idea of the important topics covered. Xu had an insider's view of the Students for Fair Admission v. Harvard trial, now awaiting appeal to the Supreme Court. He also describes the successful effort by a grass roots coalition of Asian Americans to defeat CA Proposition 16, which would have reinstated racial preferences in the public sector (including college admissions) which were prohibited by Proposition 209 in 1996.

Over the years I have had many conversations on this topic with well-meaning (but often poorly informed) parents of all ethnic and cultural backgrounds. I cannot help but ask these people

Are you OK with discrimination against your child? What did they do to deserve it? 

Are you going to let virtue-signaling administrators at the university devalue the hard work and hard-won accomplishments of your son or daughter? Are you going to do anything about it?

and I cannot help but think

If you won't do anything about it, then f*ck you. Your kids deserve better parents.

Kenny calls it a Fight for Meritocracy. That's what it is -- a fight. Don't forget that Meritocracy is just a fancy word for fairness. It's a fight for your kid, and all kids, to be treated fairly.

I highly recommend the book. These issues are of special concern to Asian Americans, but should be of interest to anyone who wants to know what is really happening in American education today.

Related posts: discrimination against Asian Americans at elite US universities, on meritocracy, and UC faculty report on the use of SAT in admissions.

Genomic prediction of student flow through high school math curriculum

Compute polygenic EA scores for 3000 US high school students of European ancestry. Track individual progress from 9th to 12th grade, focusing on mathematics courses. The students are out-of-sample: not used in training of predictor. In fact, a big portion (over half?) of individuals used in predictor training are not even from the US -- they are from the UK/EU.

Results: predictor captures about as much variance as family background (SES = Social Economic Status). Students with lower polygenic scores are less likely to take advanced math (e.g., Geometry and beyond).

Typical education paths of individuals with, e.g., bottom few percentile polygenic score are radically different from those in the top percentiles, even after controlling for SES. For example, consider only rich kids or kids at superior schools and compare educational trajectory vs polygenic score. Looks like (bottom figure) odds ratio for taking Geometry in 9th grade is about 4-6x higher for top polygenic score kids.

Genetic Associations with Mathematics Tracking and Persistence in Secondary School

K. Paige Harden and Benjamin W. Domingue, et al.

...we address this question using student polygenic scores, which are DNA-based indicators of propensity to succeed in education8. We integrated genetic and official school transcript data from over 3,000 European-ancestry students from U.S. high schools. We used polygenic scores as a molecular tracer to understand how the flow of students through the high school math pipeline differs in socioeconomically advantaged versus disadvantaged schools. Students with higher education polygenic scores were tracked to more advanced math already at the beginning of high school and persisted in math for more years...

...including family-SES and school-SES as covariates attenuated the association between the education-PGS and mathematics tracking in the 9th-grade only by about 20% (attenuated from b = 0.583, SE = .034, to b = 0.461, SE = .036, p < 2 × 10-16, Supplementary Table S3). Note that the association with genetics was roughly comparable in magnitude to the association with familySES...

See also Game Over: Genomic Prediction of Social Mobility (some overlap in authors with the new paper).

A talk by the first author:

New Statesman: "like it or not, the debate about whether genes affect intelligence is over"

Science writer Philip Ball, a longtime editor at Nature, writes a sensible article about the implications of rapidly improving genomic prediction for cognitive ability.

Philip Ball is a freelance science writer. He worked previously at Nature for over 20 years, first as an editor for physical sciences (for which his brief extended from biochemistry to quantum physics and materials science) and then as a Consultant Editor. His writings on science for the popular press have covered topical issues ranging from cosmology to the future of molecular biology.

Philip is the author of many popular books on science, including works on the nature of water, pattern formation in the natural world, colour in art, the science of social and political philosophy, the cognition of music, and physics in Nazi Germany.

... Philip has a BA in Chemistry from the University of Oxford and a PhD in Physics from the University of Bristol.

I recommend the whole article -- perhaps it will stimulate a badly needed discussion of this rapidly advancing area of science.

The IQ trap: how the study of genetics could transform education (New Statesman)

The study of the genes which affect intelligence could revolutionise education. But, haunted by the spectre of eugenics, the science risks being lost in a political battle.

... Researchers are now becoming confident enough to claim that the information available from sequencing a person’s genome – the instructions encoded in our DNA that influence our physical and behavioural traits – can be used to make predictions about their potential to achieve academic success. “The speed of this research has surprised me,” says the psychologist Kathryn Asbury of the University of York, “and I think that it is probable that pretty soon someone – probably a commercial company – will start to try to sell it in some way.” Asbury believes “it is vital that we have regulations in place for the use of genetic information in education and that we prepare legal, social and ethical cases for how it could and should be used.”

... Some kids pick things up in a flash, others struggle with the basics. This doesn’t mean it’s all in their genes: no one researching genes and intelligence denies that a child’s environment can play a big role in educational attainment. Of course kids with supportive, stimulating families and motivated peers have an advantage, while in some extreme cases the effects of trauma or malnutrition can compromise brain development.

... Robert Plomin of King’s College London, one of the leading experts on the genetic basis of intelligence, and his colleague Sheila Walker. They surveyed almost 2,000 primary school teachers and parents about their perceptions of genetic influence on a number of traits, including intelligence, and found that on the whole, both teachers and parents rated genetics as being just as important as the environment. This was despite the fact that 80 per cent of the teachers said there was no mention of genetics in their training. Plomin and Walker concluded that educators do seem to accept that genes influence intelligence.

Kathryn Asbury supports that view. When her PhD student Madeline Crosswaite investigated teachers’ beliefs about intelligence, Asbury says she found that “teachers, on average, believe that genetic factors are at least as important as environmental factors” and say they are “open to a role for genetic information in education one day, and that they would like to know more”.

... But now it’s possible to look directly at people’s genomes: to read the molecular code (sequence) of large proportions of an individual’s DNA. Over the past decade the cost of genome sequencing has fallen sharply, making it possible to look more directly at how genes correlate with intelligence. The data both from twin studies and DNA analysis are unambiguous: intelligence is strongly heritable. Typically around 50 per cent of variations in intelligence between individuals can be ascribed to genes, although these gene-induced differences become markedly more apparent as we age. As Ritchie says: like it or not, the debate about whether genes affect intelligence is over.

... Genome-wide polygenic scores can now be used to make such predictions about intelligence. They’re not really reliable at the moment, but will surely become better as the sample sizes for genome-wide studies increase. They will always be about probabilities, though: “Mrs Larkin, there is a 67 per cent chance that your son will be capable of reaching the top 10 per cent of GCSE grades.” Such exam results were indeed the measure Plomin and colleagues used for one recent study of genome-based prediction. They found that there was a stronger correlation between GPS and GCSE results for extreme outcomes – for particularly high or low marks.

... Using GPSs from nearly 5,000 pupils, the report assesses how exam results from different types of school – non-selective state, selective state grammar, and private – are correlated with gene-based estimates of ability for the different pupil sets. The results might offer pause for thought among parents stumping up eyewatering school fees: the distribution of exam results at age 16 could be almost wholly explained by heritable differences, with less than 1 per cent being due to the type of schooling received. In other words, as far as academic achievement is concerned, selective schools seem to add next to nothing to the inherent abilities of their pupils. ...

Low SES does not decrease heritability of cognitive ability (N=300k)

These researchers, from Stanford, Northwestern, and the University of Florida, analyze a large population of twins and siblings (~24k twins and ~300k children in total, born 1994-2002 in Florida). They find no evidence of SES (Socio-Economic Status) moderation of genetic influence on test scores (i.e., cognitive ability). The figure above shows the usual pattern of lower pairwise correlations in test performance between non-identical twins and ordinary sibs, consistent with strong heritability. (In figure, ICC = Intraclass Correlation = ratio of between-pair variance to total variance; SS/OS = Same/Opposite Sex.) The researchers find, via further analysis (see below), that lower SES does not decrease heritability. No large GxE effect at low SES.

Earlier work by Turkheimer and collaborators (with much smaller sample size) suggested that low SES can drastically reduce the genetic heritability of intelligence. Their result has been widely publicized, but over time evidence is accumulating against it.

Note that Economics Nobelist James J. Heckman is the editor at PNAS who handled this paper. Heckman is an expert statistician and one of the most highly cited researchers in the area of childhood education and human capital. He was also a vocal critic of The Bell Curve, but seems (now) to accept the validity of general intelligence as a construct, its heritability, and the difficulty of increasing intelligence through environmental intervention. He tends to focus on other, more trainable, factors that influence life success, such as (my interpretation) Conscientiousness, Rule Following, Pro-Sociality, etc. ("non-cognitive skills").

Socioeconomic status and genetic influences on cognitive development
PNAS doi: 10.1073/pnas.1708491114

Significance
A prominent hypothesis in the study of intelligence is that genetic influences on cognitive abilities are larger for children raised in more advantaged environments. Evidence to date has been mixed, with some indication that the hypothesized pattern may hold in the United States but not elsewhere. We conducted the largest study to date using matched birth and school administrative records from the socioeconomically diverse state of Florida, and we did not find evidence for the hypothesis.

Abstract
Accurate understanding of environmental moderation of genetic influences is vital to advancing the science of cognitive development as well as for designing interventions. One widely reported idea is increasing genetic influence on cognition for children raised in higher socioeconomic status (SES) families, including recent proposals that the pattern is a particularly US phenomenon. We used matched birth and school records from Florida siblings and twins born in 1994–2002 to provide the largest, most population-diverse consideration of this hypothesis to date. We found no evidence of SES moderation of genetic influence on test scores, suggesting that articulating gene-environment interactions for cognition is more complex and elusive than previously supposed.

From the paper. Note SS/OS = Same/Opposite Sex, SES = Socio-Economic Status.

First, Turkheimer and Horn indicate that “the between-pair variance of MZ pairs decreases in poor environments” (ref. 21, p. 63). Contrary to this relationship, we found that the between-pair variance of SS twins is actually lowest in the highest SES families. Given that SS twins are a relatively equal combination of MZ and DZ twins, one possibility is that a pattern supporting the hypothesis among MZ SS twins is masked by an even stronger pattern in the opposite direction among DZ SS twins. However, Fig. 3 shows that corresponding results for OS twins (all of whom are DZ) give no indication of such a pattern. Between-pair variances in achievement test scores for high-school educated parents of OS twins are higher in all cases than it is for parents without a high school diploma.

Second, Turkheimer and Horn report that “the within-pair variance of MZ twin pairs increases at lower levels of SES: poverty appears to have the effect of making MZ twins more different from each other” (ref. 21, p. 61). We would therefore expect in our data that the within-pair variance for SS twins whose mother did not graduate from high school would be higher than the variance for SS twins whose mother has a high school diploma. However, this is not the case in any of the SS twin comparisons shown in Fig. 3.

Via SSC -- thanks, Scott!

Added remarks about context and broader implications: This paper does not exclude SES effects on intelligence. Rather, it excludes a hypothesis (big nonlinear effect at low SES; GxE!) that has been widely discussed: In good environments individuals can achieve their full genetic potential, and consequently measured heritability is high. However, in bad environments individuals don't achieve their full genetic potential, and (perhaps) do not even realize the full effect of beneficial genetic variants, so heritability is much reduced. This reasonable sounding hypothesis is not supported by the Florida data, suggesting that genetic influence is similarly strong in both high and low SES families.

Now, just how strong is this genetic influence? Many large studies have been conducted on populations of twins (raised together and apart), adoptees (who end up resembling their biological parents much more than the adoptive parents who raised them), and ordinary siblings. The results suggest very high heritability of adult intelligence -- broad sense heritability may be as high as ~0.8!

Wikipedia: Recent twin and adoption studies suggest that while the effect of the shared family environment is substantial in early childhood, it becomes quite small by late adolescence. These findings suggest that differences in the life styles of families whatever their importance may be for many aspects of children's lives make little long-term difference for the skills measured by intelligence tests.

Python Programming in one video

Putting this here in hopes I can get my kids to watch it at some point 8-)

Please recommend similar resources in the comments!

The mystery of genius at Slate Star Codex

Three excellent posts at Slate Star Codex. Don't miss the comments -- there are over a thousand, many of them very good.

THE ATOMIC BOMB CONSIDERED AS HUNGARIAN HIGH SCHOOL SCIENCE FAIR PROJECT

A group of Manhattan Project physicists created a tongue-in-cheek mythology where superintelligent Martian scouts landed in Budapest in the late 19th century and stayed for about a generation, after which they decided the planet was unsuitable for their needs and disappeared. The only clue to their existence were the children they had with local women.

The joke was that this explained why the Manhattan Project was led by a group of Hungarian supergeniuses, all born in Budapest between 1890 and 1920. These included Manhattan Project founder Leo Szilard, H-bomb creator Edward Teller, Nobel-Prize-winning quantum physicist Eugene Wigner, and legendary polymath John von Neumann, namesake of the List Of Things Named After John Von Neumann.

The coincidences actually pile up beyond this. Von Neumann, Wigner, and possibly Teller all went to the same central Budapest high school at about the same time, leading a friend to joke about the atomic bomb being basically a Hungarian high school science fair project. ...

See also

HUNGARIAN EDUCATION II: FOUR NOBEL TRUTHS

and

HUNGARIAN EDUCATION III: MASTERING THE CORE TEACHINGS OF THE BUDAPESTIANS

... Laszlo Polgar studied intelligence in university, and decided he had discovered the basic principles behind raising any child to be a genius. He wrote a book called Bring Up Genius and recruited an interested woman to marry him so they could test his philosophy by raising children together. He said a bunch of stuff on how ‘natural talent’ was meaningless and so any child could become a prodigy with the right upbringing.

This is normally the point where I’d start making fun of him. Except that when he trained his three daughters in chess, they became the 1st, 2nd, and 6th best female chess players in the world, gaining honors like “youngest grandmaster ever” and “greatest female chess player of all time”. Also they spoke seven languages, including Esperanto.

Their immense success suggests that education can have a major effect even on such traditional genius-requiring domains as chess ability. How can we reconcile that with the rest of our picture of the world, and how obsessed should we be with getting a copy of Laszlo Polgar’s book? ...

Robots Proctor Online Exams

For background on this subject, see How to beat online exam proctoring. It is easy for clever students to beat existing security systems for online exams. Enterprising students could even set up "cheating rooms" that make it easy for test takers to cheat. Judging by the amount of traffic this old post gets, cheating on online exams is a serious problem.

Machine learning to the rescue! :-) The machines don't have to be 100% accurate in detection -- they can merely flag suspicious moments in the data and ask a human proctor to look more carefully. This makes the overall system much more scalable.

The monitoring data (e.g., video from webcam + pov cam) from a particular exam could potentially be stored forever. In an extreme case, a potential employer who wants to be sure that Johnny passed the Python coding (or psychometric g) exam for real could be granted access to the stored data by Johnny, to see for themselves.

Automated Online Exam Proctoring
Atoum, Chen, Liu, Hsu, and Liu
IEEE Transactions on Multimedia

Abstract:
Massive open online courses (MOOCs) and other forms of remote education continue to increase in popularity and reach. The ability to efficiently proctor remote online examinations is an important limiting factor to the scalability of this next stage in education. Presently, human proctoring is the most common approach of evaluation, by either requiring the test taker to visit an examination center, or by monitoring them visually and acoustically during exams via a webcam. However, such methods are labor-intensive and costly. In this paper, we present a multimedia analytics system that performs automatic online exam proctoring. The system hardware includes one webcam, one wearcam, and a microphone, for the purpose of monitoring the visual and acoustic environment of the testing location. The system includes six basic components that continuously estimate the key behavior cues: user verification, text detection, voice detection, active window detection, gaze estimation and phone detection. By combining the continuous estimation components, and applying a temporal sliding window, we design higher level features to classify whether the test taker is cheating at any moment during the exam. To evaluate our proposed system, we collect multimedia (audio and visual) data from 24 subjects performing various types of cheating while taking online exams. Extensive experimental results demonstrate the accuracy, robustness, and efficiency of our online exam proctoring system.

This work is related to the issued patent

Online examination proctoring system
WO 2014130769 A1

ABSTRACT
The system to proctor an examination includes a first camera (10) worn by the examination taking subject (12) and directed to capture images in subject's field of vision. A second camera (14) is positioned to record an image of the subject's face during the examination. A microphone (26) captures sounds within the room, which are analyzed to detect speech utterances. The computer system (8) is programmed to store captured images from said first camera. The computer (16) is also programmed to issue prompting events instructing the subject to look in a direction specified by the computer at event intervals not disclosed to subject in advance and to index for analysis the captured images in association with indicia corresponding to the prompting events.

Publication number WO2014130769 A1
Publication type Application
Application number PCT/US2014/017584
Publication date Aug 28, 2014
Filing date Feb 21, 2014
Priority date Feb 25, 2013
Also published as US9154748, US20140240507
Inventors Stephen Hsu, Xiaoming Liu, Xiangyang Alexander LIU
Applicant Board Of Trustees Of Michigan State University

Genetics, Cognitive Ability, and Education (conversation with Cambridge PhD candidate Daphne Martschenko)

Further conversation with Cambridge PhD candidate Daphne Martschenko concerning genetics of cognitive ability, implications for education policy, etc.

See also earlier conversation: https://www.youtube.com/watch?v=YVqkvHpLfuQ

Dunedin paper referenced in the video (polygenic score prediction of adult success for different SES groups): http://infoproc.blogspot.com/2016/09/genomic-prediction-of-adult-life.html

Visualization of geometric intuitions underlying linear algebra (video)

Meritocracy and DNA

Read Toby Young's new article The Fall of the Meritocracy. The adoption of civil service and educational placement examinations in England and France in the 19th century was consciously based on the Chinese model (see Les Grandes Ecoles Chinoises). Although imperfect, exams work: we have no better system for filtering talent. But few are willing to acknowledge this basic fact.

Quadrant Magazine: In 1958, my father, Michael Young, published a short book called The Rise of the Meritocracy, 1870–2023: An Essay on Education and Equality. It purported to be a paper written by a sociologist in 2034 about the transformation of Britain from a feudal society in which people’s social position and level of income were largely determined by the socio-economic status of their parents into a modern Shangri-La in which status is based solely on merit. He invented the word meritocracy to describe this principle for allocating wealth and prestige and the new society it gave rise to.

The essay begins with the introduction of open examinations for entry into the civil service in the 1870s—hailed as “the beginning of the modern era”—and continues to discuss real events up until the late 1950s, at which point it veers off into fantasy, describing the emergence of a fully-fledged meritocracy in Britain in the second half of the twentieth century. In spite of being semi-fictional, the book is clearly intended to be prophetic—or, rather, a warning. Like George Orwell’s Nineteen Eighty-Four (1949), The Rise of the Meritocracy is a dystopian satire that identifies various aspects of the contemporary world and describes a future they might lead to if left unchallenged. Michael was particularly concerned about the introduction of the 11+ by Britain’s wartime coalition government in 1944, an intelligence test that was used to determine which children should go to grammar schools (the top 15 per cent) and which to secondary moderns and technical schools (the remaining 85 per cent). It wasn’t just the sorting of children into sheep and goats at the age of eleven that my father objected to. As a socialist, he disapproved of equality of opportunity on the grounds that it gave the appearance of fairness to the massive inequalities created by capitalism. He feared that the meritocratic principle would help to legitimise the pyramid-like structure of British society.

In the short term, the book achieved its political aim. It was widely read by Michael’s colleagues in the Labour Party (he ran the party’s research department from 1945 to 1951) and helped persuade his friend Anthony Crosland, who became Labour Education Secretary in 1965, that the 11+ should be phased out and the different types of school created by the 1944 Education Act should be replaced by non-selective, one-size-fits-all comprehensives. Crosland famously declared: “If it’s the last thing I do, I’m going to destroy every f***ing grammar school in England. And Wales and Northern Ireland.” Today, there are only 164 grammar schools in England and sixty-eight in Northern Ireland. There are none in Wales.

But even though my father’s book helped to win the battle over selective education, he lost the war. The term “meritocracy” has now entered the language, and while its meaning hasn’t changed—it is still used to describe the organising principle Michael identified in his book—it has come to be seen as something good rather than bad.[1] The debate about grammar schools rumbles on in Britain, but their opponents no longer argue that a society in which status is determined by merit is undesirable. Rather, they embrace this principle and claim that a universal comprehensive system will lead to higher levels of social mobility than a system that allows some schools to “cream skim” the most intelligent children at the age of eleven.[2]

We are all meritocrats now

Not only do pundits and politicians on all sides claim to be meritocrats—and this is true of most developed countries, not just Britain—they also agree that the principle remains stillborn. In Britain and America there is a continuing debate about whether the rate of inter-generational social mobility has remained stagnant or declined in the past fifty years, but few think it has increased.[3] The absence of opportunities for socio-economic advancement is now seen as one of the key political problems facing Western democracies, leading to the moral collapse of the indigenous white working class, the alienation of economically unsuccessful migrant groups, and unsustainable levels of welfare dependency. This cluster of issues is the subject of several recent books by prominent political scientists, most notably Our Kids: The American Dream in Crisis (2015) by Robert Putnam.

Unlike my father, I’m not an egalitarian. As Friedrich Hayek and others have pointed out, the difficulty with end-state equality is that it can only be achieved at too great a human cost. Left to their own devices, some men will inevitably accumulate more wealth than others, whether through ability or luck, and the only way to “correct” this is through the state’s use of coercive power. If the history of the twentieth century teaches us anything, it is that the dream of creating a socialist utopia often leads to the suppression of free speech, the imprisonment of a significant percentage of the population and, in some extreme cases, state-organised mass murder.

Having said that, I recognise that a lack of social mobility poses a threat to the sustainability of liberal democracies and, in common with many others, believe the solution lies in improving our education systems. There is a consensus among most participants in the debate about education reform that the ideal schools are those that manage to eliminate the attainment gap between the children of the rich and the poor. That is, an education system in which children’s exam results don’t vary according to the neighbourhood they’ve grown up in, the income or education of their parents, or the number of books in the family home. Interestingly, there is a reluctance on the part of many liberal educationalists to accept the corollary of this, which is that attainment in these ideal schools would correspond much more strongly with children’s natural abilities. [4] This is partly because it doesn’t sit well with their egalitarian instincts and partly because they reject the idea that intelligence has a genetic basis. But I’m less troubled by this. I want the clever, hard-working children of those in the bottom half of income distribution to move up, and the less able children of those in the top half to move down. ...

Young follows the meritocracy and social justice argument where it ultimately leads: to DNA.

... However, there’s a problem here—let’s call it the challenge posed by behavioural genetics—which is that cognitive ability and other characteristics that lead to success, such as conscientiousness, impulse control and a willingness to defer gratification, are between 40 per cent and 80 per cent heritable.[5] I know that many people will be reluctant to accept that, but the evidence from numerous studies of identical twins separated at birth, as well as non-biological siblings raised in the same household, is pretty overwhelming. And it’s probable that in the next few years genetic research scientists will produce even more evidence that important aspects of people’s personalities—including those that determine whether they succeed or fail—are linked to their genes, with the relevant variants being physically identified. The implication is that a society in which status is allocated according to merit isn’t much fairer than one in which it’s inherited—or, rather, it is partly inherited, but via parental DNA rather than tax-efficient trusts. This is an argument against meritocracy made by John Rawls in A Theory of Justice (1971): You’ve done nothing to deserve the talents you’re born with—they’re distributed according to a “natural lottery”—so you don’t deserve what flows from them.[6]

It’s worth pausing here to note that Rawls accepts that not all men are born equal, genetically speaking. Some do better out of the “natural lottery” than others and that, in turn, has an impact on their life chances. This is far from universally accepted by liberal commentators and policy-makers, most of whom prefer to think of man as a tabula rasa, forged by society rather than nature. ...

... The reason liberals are so hostile to the concept of IQ—and particularly the claim that it helps to determine socio-economic status, rather than vice versa—is because they have an almost religious attachment to the idea that man is a piece of clay that can be moulded into any shape by society. After all, it’s only if human beings are infinitely malleable and not bound by their inner nature that the various utopias they dream of can become a reality, from William Morris’s Earthly Paradise to the New Jerusalem of my father’s Labour Party. ...

... But the new technologies thrown up by genetic research will mean they no longer have to deny this obvious truth. If it becomes possible to select human embryos according to their possession of genes associated with certain character traits, such as intelligence, the Left’s utopian political projects can be resurrected. Margaret Mead was right after all: human nature is almost unbelievably malleable, you just have to start a lot further back. It is not through changing the culture that we will be able to solve the chronic social problems besetting the advanced societies of the West, but through changing people’s genes. ...

Education and Achievement Gaps

This recent talk by Harvard economist and education researcher Roland Fryer reviews studies of student incentives, charter schools, best educational practices, and their effects on achievement gaps.  Audio  Slides (the features in the image below are not clickable).

A very recent preprint on a study of parental incentives:

Parental Incentives and Early Childhood Achievement: A Field Experiment in Chicago Heights

Roland G. Fryer, Jr.
Harvard University and NBER

Steven D. Levitt
University of Chicago and NBER

John A. List
University of Chicago and NBER

August 2015

Abstract
This article describes a randomized field experiment in which parents were provided financial incentives to engage in behaviors designed to increase early childhood cognitive and executive function skills through a parent academy. Parents were rewarded for attendance at early childhood sessions, completing homework assignments with their children, and for their child’s demonstration of mastery on interim assessments. This intervention had large and statistically significant positive impacts on both cognitive and non-cognitive test scores of Hispanics and Whites, but no impact on Blacks. These differential outcomes across races are not attributable to differences in observable characteristics (e.g. family size, mother’s age, mother’s education) or to the intensity of engagement with the program. Children with above median (pre-treatment) non cognitive scores accrue the most benefits from treatment.

Seasons and Veritas

Harvard graduates explain why we have seasons. If only their understanding matched their confidence.

See also Why is it dark at night?  ,  Inside HBS: "kill, f^^k or marry"  ,  Frauds!  and

High V, Low M: ... high verbal ability is useful for appearing to be smart, or for winning arguments and impressing other people, but it's really high math ability that is useful for discovering things about the world -- that is, discovering truth or reasoning rigorously.

Measuring college learning outcomes: psychometry 101

Pressure is growing for outcomes testing in higher education. Already hundreds of schools allow graduating seniors to take the CLA+ (Collegiate Learning Assessment Plus) to provide evidence of important job skills. I doubt that the CLA+ adds much information concerning an applicant's abilities beyond what can be obtained from existing cognitive tests such as SAT, ACT, GRE. But those tests have plenty of enemies, creating a business opportunity for shiny new assessments. The results covered below will contain no surprises to anyone modestly familiar with modern psychometrics.

Forbes: More people than ever are asking the question: is college worth it? Take a look at the numbers: 81 percent of respondents in a 2008 survey of the general public agreed that a college education was a good investment, but that number was down to 57 percent in 2012. A recent Wells Fargo study reported that one-third of Millennials regret going to college, and instead say they would have been better off working and earning money. This rhetoric is reflected in the reality of declining enrollment: one survey of colleges showed that enrollment for spring 2013 was down 2.3 percent from spring 2012, a trend that has held for consecutive years.

Meanwhile, the wave of keeping colleges accountable for their outcomes continues to crest, even from the left. A recent Brookings Institution study concluded, “While the average return to obtaining a college degree is clearly positive, we emphasize that it is not universally so.” President Obama, a major recipient of plaudits and campaign dollars from the academic left, has called for a government-authored set of rankings for American colleges and universities that rewards performance and punishes failure: “It is time to stop subsidizing schools that are not producing good results, and reward schools that deliver for American students and our future,” he said.

President Obama’s impulse to define and reward value in higher education was correct, but a government-rankings system is not a sufficient corrective for the enormity of the problem. There is no panacea for reforming higher education, but the CLA+ exam has potential to be a very useful step. ...

More from the Wall Street Journal.

WSJ: A survey of business owners to be released next week by the American Association Colleges and Universities also found that nine out of 10 employers judge recent college graduates as poorly prepared for the work force in such areas as critical thinking, communication and problem solving.

“Employers are saying I don’t care about all the knowledge you learned because it’s going to be out of date two minutes after you graduate ... I care about whether you can continue to learn over time and solve complex problems,” said Debra Humphreys, vice president for policy and public engagement at AAC&U, which represents more than 1,300 schools.

The CLA+ [Collegiate Learning Assessment Plus] is graded on a scale of 400 to 1600. In the fall of 2013, freshmen averaged a score of 1039, and graduating seniors averaged 1128, a gain of 89 points.

CAE says that improvement is evidence of the worth of a degree. “Colleges and universities are contributing considerably to the development of key skills that can make graduates stand out in a competitive labor market,” the report said.

Mr. Arum was skeptical of the advantages accrued. Because the test was administered over one academic year, it was taken by two groups of people. A total of 18,178 freshmen took the test and 13,474 seniors. That mismatch suggested a selection bias to Mr. Arum.

“Who knows how many dropped out? They were probably the weaker students,” he said. [ THIS LAST POINT IS ADDRESSED BELOW. SCORE GAINS AFTER ADJUSTMENT FOR THIS EFFECT ARE MODEST -- TYPICALLY LESS THAN 0.5 SD; SOMETIMES CONSISTENT WITH ZERO. ]

What exactly are these college learning assessments? They measure general skills that employers deem important, but not narrow subject matter expertise -- some of which is economically valuable (e.g., C++ coding) and some much less so (e.g., mastery of ancient Greek or the works of Shakespeare). Of course, narrow job-essential knowledge can be tested separately.

What Does CLA+ Measure?

CLA+ [Collegiate Learning Assessment Plus] is designed specifically to measure critical-thinking and written-communication skills that other assessments cannot. CAE has found that these are the skills that most accurately attest to a student’s readiness to enter the workforce. In the era of Google, the ability to recall facts and data is not as crucial as it once was. As our technology evolves to meet certain needs of the workplace, so too must our thinking about success and career readiness. Therefore, the skills taught in higher education are changing; less emphasis is placed on content-specific knowledge, and more is placed on critical-thinking skills, such as scientific and quantitative reasoning, analysis and problem solving, and writing effectiveness and mechanics. That is why CLA+ focuses on these skills and why CAE believes employers should use this tool during recruitment efforts.

Sample.

Two important questions:

1) Are the CLA+ and related assessments measuring something other than the general cognitive ability of individuals who have had many years (K-12 plus at least some college) of education?

2) By how much does a college education improve CLA+ scores?

The study below, which involved 13 colleges (ranging from MIT, Michigan, Minnesota, to Cal-State Northridge, Alabama A&M, ...) gives some hints at answers.

Test Validity Study (TVS) Report

This study examined whether commonly used measures of college-level general educational outcomes provide comparable information about student learning. Specifically, do the students and schools earning high scores on one such test also tend to earn high scores on other tests designed to assess the same or different skills? And, are the strengths of these relationships related to the particular tests used, the skills (or “constructs”) these tests are designed to measure (e.g., critical thinking, mathematics, or writing), the format they use to assess these skills (multiple-choice or constructed-response), or the tests’ publishers? We also investigated whether the difference in mean scores between freshmen and seniors was larger on some tests than on others. Finally, we estimated the reliability of the school mean scores on each measure to assess the confidence that can be placed in the test results.

Effect sizes are modest. The result "d+" in the table below is the average increase in score between freshmen and seniors tested, in units of standard deviations. An individual's score as a freshman is probably a very good predictor of their score as a senior. (To put it crudely, additional years of expensive post-secondary education do not increase cognitive ability by very much. What cognitive tests measure is fairly stable, despite the efforts of educators.)

Note, in order to correct for the problem that weaker students drop out between freshman and senior years, and hence the senior population is academically stronger, the researchers adjusted effect sizes. The adjustment used was simply the average SAT score difference (in SD units) between seniors and freshmen in each school's sample (students who survive to senior year tend to have higher SAT scores -- go figure!). In other words, to get their final results, the researchers implicitly acknowledged that these new tests are largely measuring the same general cognitive abilities as the SAT!

Below are school-level correlations and reliabilities on various assessments, which show that cognitive constructs ("critical thinking", "mathematics", etc.) are consistently evaluated regardless of specific test used. Hint: ACT, SAT, GRE, PISA, etc. would have worked just as well ...

The results below are also good evidence for a school-level general factor of ability = "G". The researchers don't release specific numbers, but I'd guess MIT has a much higher G than some of the lower ranked schools, and that the value of G can be deduced just from average SAT score of the school.

Does the CLA have validity in predicting job and life outcomes? Again, experienced psychometricians know the answer, but stay tuned as data gradually accumulate.

Documenting Uncertain Times: Post-graduate Transitions of the Academically Adrift Cohort

Graduates who scored in the bottom quintile of the CLA were three times more likely to be unemployed than those who scored in the top quintile on the CLA (9.6 percent compared to 3.1 percent), twice as likely to be living at home (35 percent compared to 18 percent) and significantly more likely to have amassed credit card debt (51 percent compared to 37 percent).

The Hollow Men: video and audio

Highly recommended talk by Dominic Cummings at the UK Institute for Public Policy Research. See also The Hollow Men (original essay).

Podcast version here and below.

The Hollow Men: What's wrong with Westminster and Whitehall, and what to do about it by Ippr on Mixcloud

Ability, Effort, and Academic Achievement among Asian Americans

What accounts for the academic success of E. Asians? I would have guessed about equal parts cognitive advantage (ability) and hard work (grinding). The paper below tries to quantify this in more detail using data from two nationally representative cohort studies, comparing students of various of ethnicities who attend the same schools. Note the broad conclusion stated in the abstract applies better to Asian Americans (AAs) in aggregate, and less well to the E. Asian subpopulation alone -- see figures below.

Explaining Asian Americans’ academic advantage over whites (PNAS)

The superior academic achievement of Asian Americans is a well-documented phenomenon that lacks a widely accepted explanation. Asian Americans’ advantage in this respect has been attributed to three groups of factors: (i) socio-demographic characteristics, (ii) cognitive ability, and (iii) academic effort as measured by characteristics such as attentiveness and work ethic. We combine data from two nationally representative cohort longitudinal surveys to compare Asian-American and white students in their educational trajectories from kindergarten through high school. We find that the Asian-American educational advantage is attributable mainly to Asian students exerting greater academic effort and not to advantages in tested cognitive abilities or socio-demographics. We test explanations for the Asian–white gap in academic effort and find that the gap can be further attributed to (i) cultural differences in beliefs regarding the connection between effort and achievement and (ii) immigration status. Finally, we highlight the potential psychological and social costs associated with Asian-American achievement success.

While all four AA subpopulations showed positive differences relative to white students in academic achievement and effort, only the E. Asian subgroup had a cognitive advantage.

From the Supplement: for E. Asians, the academic achievement gap appears to be ~ 1/3 due to cognitive ability and ~ 2/3 due to academic effort, with large uncertainties. For the other subpopulations the results are quite different (and actually rather strange). Click for larger version.

Income, Cognitive Ability, and Education

The figure below is from Schooling, Intelligence, and Income (1997, American Psychologist) by Ceci and Williams.

Cognitive ability has a strong impact on earnings at each level of educational attainment.

Don't forget to respond to the reader survey.

Intervention and Replication

This article describes surprisingly effective interventions that improve college success rates for low SES/SAT students. Let's hope they can be broadly replicated.

NYTimes: ... Laude was hopeful that the small classes would make a difference, but he recognized that small classes alone wouldn’t overcome that 200-point SAT gap. “We weren’t naïve enough to think they were just going to show up and start getting A’s, unless we overwhelmed them with the kind of support that would make it possible for them to be successful,” he said. So he supplemented his lectures with a variety of strategies: He offered TIP students two hours each week of extra instruction; he assigned them advisers who kept in close contact with them and intervened if the students ran into trouble or fell behind; he found upperclassmen to work with the TIP students one on one, as peer mentors. And he did everything he could, both in his lectures and outside the classroom, to convey to the TIP students a new sense of identity: They weren’t subpar students who needed help; they were part of a community of high-achieving scholars.

Even Laude was surprised by how effectively TIP worked. “When I started giving them the tests, they got the same grades as the larger section,” he said. “And when the course was over, this group of students who were 200 points lower on the SAT had exactly the same grades as the students in the larger section.” The impact went beyond Chemistry 301. This cohort of students who, statistically, were on track to fail returned for their sophomore year at rates above average for the university as a whole, and three years later they had graduation rates that were also above the U.T. average.

... Laude wanted something that would help him predict, for any given incoming freshman, how likely he or she would be to graduate in four years. The Institutional Research team analyzed the performance of tens of thousands of recent U.T. students, and from that analysis they produced a tool they called the Dashboard — an algorithm, in spreadsheet form, that would consider 14 variables, from an incoming student’s family income to his SAT score to his class rank to his parents’ educational background, and then immediately spit out a probability, to the second decimal place, of how likely he was to graduate in four years.

In the spring of 2013, Laude and his staff sat down with the Dashboard to analyze the 7,200 high-school seniors who had just been admitted to the class of 2017. When they ran the students’ data, the Dashboard indicated that 1,200 of them — including Vanessa Brewer — had less than a 40 percent chance of graduation on time. Those were the kids Laude decided to target. He assigned them each to one or more newly created or expanded interventions. The heart of the project is a portfolio of “student success programs,” each one tailored, to a certain extent, for a different college at U.T. — natural sciences, liberal arts, engineering — but all of them following the basic TIP model Laude dreamed up 15 years ago: small classes, peer mentoring, extra tutoring help, engaged faculty advisers and community-building exercises.

... Students were randomly sorted into four categories. A “belonging” treatment group read messages from current students explaining that they felt alone and excluded when they arrived on campus, but then realized that everyone felt that way and eventually began to feel at home. A “mind-set” treatment group read an article about the malleability of the brain and how practice makes it grow new connections, and then read messages from current students stating that when they arrived at U.T., they worried about not being smart enough, but then learned that when they studied they grew smarter. A combination treatment group received a hybrid of the belonging and mind-set presentations. And finally, a control group read fairly banal reflections from current students stating that they were surprised by Austin’s culture and weather when they first arrived, but eventually they got used to them. Students in each group were asked, after clicking through a series of a dozen or so web pages, to write their own reflections on what they’d read in order to help future students. The whole intervention took between 25 and 45 minutes for students to complete, and more than 90 percent of the incoming class completed it.

... In January 2013, when Yeager analyzed the first-semester data, he saw the advantaged students’ results were exactly the same as they were every year. No matter which message they saw in the pre-orientation presentation, 90 percent of that group was on track. Similarly, the disadvantaged students in the control group, who saw the bland message about adjusting to Austin’s culture and weather, did the same as disadvantaged students usually did: 82 percent were on track. But the disadvantaged students who had experienced the belonging and mind-set messages did significantly better: 86 percent of them had completed 12 credits or more by Christmas. They had cut the gap between themselves and the advantaged students in half.

One problem with this type of treatment or intervention is that the selected group should not be stigmatized, or even aware of why they were selected for the special treatment. As awareness grows (e.g., through this high profile article in the NYTimes magazine), it becomes harder and harder to camouflage the criteria used to select students for programs like TIP. Note, in the last intervention described above it appears the entire freshman class (~ 7k students) at UT Austin was used, so sample size was fairly large.