Showing posts with label psychometrics. Show all posts
Showing posts with label psychometrics. Show all posts

Wednesday, December 13, 2023

PISA 2023 and the Gloomy Prospect

I'm in the Philippines now. I flew here after the semester ended, in order to meet with outsourcing (BPO = Business Process Outsourcing) companies that run call centers for global brands. This industry accounts for ~8% of Philippine GPD (~$40B per annum), driven by comparative advantages such as the widespread use of English here and relatively low wages. 

I predict that AIs of the type produced by my startup SuperFocus.ai will disrupt the BPO industry in coming years, with dramatic effects on the numbers of humans employed in areas like customer support. I was just interviewed for the podcast of the AI expert at IBPAP, the BPO trade association - he is tasked with helping local companies adopt AI technology, and adapt to a world with generative LLMs like GPT4. I'll publish a link to that interview when it goes live. 


During my visit the latest PISA results were released. This year they provided data with students grouped by Socio-Economic Status [1], so that students in different countries, but with similar levels of wealth and access to educational resources, can be compared directly. See figures below - OECD mean ~500, SD~100. 


Quintiles are defined using the *entire* international PISA student pool. These figures allow us to compare equivalent SES cohorts across countries and to project how developing countries will perform as they get richer and improve schooling.

In some countries, such as Turkey or Vietnam, the small subset of students that are in the top quintile of SES (among all PISA students tested) already score better than the OECD average for students with similar SES. On the other hand, for most developing countries, such as the Philippines, Indonesia, Saudi Arabia, Brazil, Mexico, etc. even the highest quintile SES students score similarly to or worse than the most deprived students in, e.g., Turkey, Vietnam, Japan, etc.

Note the top 20% SES quintile among all PISA takers is equivalent to roughly top ~30% SES among Japanese. If the SES variable is even crudely accurate, typical kids in this category are not deprived in any way and should be able to achieve their full cognitive potential. In developing countries only a small percentage of students are in this quintile - they are among the elites with access to good schools, nutrition, and potentially with educated parents. Thus it is very bad news that even this subgroup of students score so poorly in almost all developing countries (with exceptions like Turkey and Vietnam). It leads to gloomy projections regarding human capital, economic development, etc. in most of the developing world. 

I had not seen a similar SES analysis before this most recent PISA report. I was hoping to see data showing catch up in cognitive ability with increasing SES in developing countries. The results indicate that cognitive gaps will be very difficult to ameliorate.

In summary, the results suggest that many of these countries will not reach OECD-average levels of human capital density even if they somehow catch up in per capita GDP.

This suggests a Gloomy Prospect for development economics. Catch up in human capital density looks difficult for most developing countries, with only a few exceptions (e.g., Turkey, Vietnam, Iran, etc.).
 

Here is the obligatory US students by ancestry group vs Rest of World graph that reflects: 1. strong US spending on education (vs Rest of World) and 2. selective immigration to the US, at least for some groups.
 

Tuesday, October 10, 2023

SMPY 65: Help support the SMPY Longitudinal Study


The Study of Mathematically Precocious Youth (SMPY) needs your help to support the Age-65 phase of their unique longitudinal study. 


For decades, co-directed by David Lubinski and Camilla P. Benbow, SMPY has been a beacon of enlightenment, tracking five cohorts comprising over 5,000 remarkably gifted individuals. In doing so, we have unraveled the secrets to nurturing brilliance. However, we are confronted with a disconcerting reality: the effective methods to identify and cultivate intellectual talent are under siege, threatened by political ideology. 

Our 14-minute documentary and the 3-page feature in Nature underscore the dire need to provide our most gifted youths with the educational opportunities they deserve. They are the architects of solutions and the architects of the future itself. 

Here are some compelling longitudinal findings from SMPY's extensive research:
• Prodigies destined for eminent careers can be identified as early as age 13. 
• There is no plateau of ability; even within the top 1%, variations in mathematical, spatial, and verbal abilities profoundly impact educational, occupational, and creative outcomes. 
• The blend of specific abilities, such as mathematical, spatial, and verbal aptitudes, shapes the nature of one's accomplishments and career trajectory.



More information:

Long

Short


DONATE HERE

Indicate "Please designate this gift to Study of Mathematically Precocious Youth" in the Special Instructions.


Thursday, October 05, 2023

Yasheng Huang: China's Examination System and its impact on Politics, Economy, Innovation — Manifold #45

 

Yasheng Huang is the Epoch Foundation Professor of Global Economics and Management at the MIT Sloan School of Management. His new book is The Rise and Fall of the EAST: How Exams, Autocracy, Stability, and Technology Brought China Success, and Why They Might Lead to Its Decline. 

Steve and Yasheng discuss: 

0:00 Introduction 
1:11 From Beijing to Harvard in the 1980s 
15:29 Civil service exams and Huang's new book, "The Rise and Fall of the EAST" 
37:14 Two goals: Developing human capital and indoctrination 
48:33 Impact of the exam system 
57:04 China's innovation peak and decline 
1:12:23 Collaboration and relationship with the West 
1:21:31 How will the U.S.-China relationship evolve? 

Audio-only version, and transcript: 

Yasheng Huang at MIT 

Web site: 

Thursday, September 07, 2023

Meritocracy, SAT Scores, and Laundering Prestige at Elite Universities — Manifold #43

 

I discuss 10 key graphs related to meritocracy and university admissions. Predictive power of SATs and other factors in elite admissions decisions. College learning outcomes - what do students learn? The four paths to elite college admission. Laundering prestige at the Ivies. 

Slides: 


Audio Only and Transcript: 


CLA and college learning outcomes

Harvard Veritas: Interview with a recent graduate 

Defining Merit - Human Capital and Harvard University


Chapter markers: 

0:00 Introduction 
1:28 University of California system report and the use of SAT scores admissions 
8:04 Longitudinal study on gifted students and SAT scores (SMPY) 
12:53 Unprecedented data on earnings outcomes and SAT scores 
15:43 How SAT scores and university pedigree influence opportunities at elite firms 
17:35 Non-academic factors fail to predict student success 
20:49 Predicted earnings 
24:24 Measured benefit of Ivy Plus attendance 
28:25 CLA: 13 university study on college learning outcomes 
32:34 Does college education improve generalist skills and critical thinking? 
42:15 The composition of elite universities: 4 paths to admission 
48:12 What happened to meritocracy? 
51:48 Hard versus Soft career tracks 
54:43 Cognitive elite at Ivies vs state flagship universities 
57:11 What happened to Caltech?

Thursday, December 15, 2022

Geoffrey Miller: Evolutionary Psychology, Polyamorous Relationships, and Effective Altruism — Manifold #26

 

Geoffrey Miller is an American evolutionary psychologist, author, and a professor of psychology at the University of New Mexico. He is known for his research on sexual selection in human evolution. 


Miller's Wikipedia page.

Steve and Geoffrey discuss: 

0:00 Geoffrey Miller's background, childhood, and how he became interested in psychology 
14:44 How evolutionary psychology is perceived and where the field is going 
38:23 The value of higher education: sobering facts about retention 
49:00 Dating, pickup artists, and relationships 
1:11:27 Polyamory 
1:24:56 FTX, poly, and effective altruism 
1:34:31 AI alignment

Thursday, November 17, 2022

Abdel Abdellaoui: Genetics, Psychiatric Traits, and Educational Attainment — Manifold #24

 

Abdel Abdellaoui is Assistant Professor of Genetics in the Department of Psychiatry, Amsterdam UMC, University of Amsterdam. 

Abdel Abdellaoui is a geneticist who has been involved in a wide range of studies on psychiatric genetics, behavioral genetics, and population genetics. He is particularly interested in how collective behaviors, such as migration and mate choice, influence the genetic makeup of populations and the relationship between genetic risk factors and environmental exposures. 

Steve and Abdel discuss: 

00:00 Abdel’s background: education, family history, research career 
10:23 Abdel’s research focus: polygenic traits, geographical stratification 
21:43 Correlations across geographical regions 
33:21 Educational Attainment 
38:51 Comparisons across data sets 
44:48 Longevity 
52:04 Reaction to NIH restricting access to data on educational attainment 

Abdel Abdellaoui on Google Scholar: 
https://scholar.google.com/citations?user=hsyseKEAAAAJ&hl=en

Sunday, November 13, 2022

Smart Leftists vs Dumb Leftists

Tuesday, October 04, 2022

SAT score distributions in Michigan

The state of Michigan required all public HS seniors to take the SAT last year (~91k out of ~107k total seniors in the state). This generated an unusually representative score sample. Full report

I'm aware of this stuff because my kids attend a public HS here.

To the uninformed, the results are shocking in a number of ways. Look specifically at the top band with scores in the 1400-1600 range. These are kids who have a chance at elite university admission, based on academic merit. For calibration, the University of Michigan median SAT score is above 1400, and at top Ivies it is around 1500.


Some remarks:

1. In the top band there are many more males than females.

2. The Asian kids are hitting the ceiling on this test.

3. There are very few students from under-represented groups who score in the top band. 

4. By looking at the math score distribution (see full report) one can estimate how many students in each group are well-prepared enough to complete a rigorous STEM major -- e.g., pass calculus-based physics.

Previously I have estimated that PRC is outproducing the US in top STEM talent by a factor as large as 10x. In a decade or two the size of their highly skilled STEM workforce (e.g., top engineers, AI researchers, biotech scientists, ...) could be 10x as large as that of the US and comparable to the rest of the world, ex-China.

This is easy to understand: their base population is about 4x larger and their K12 performance on international tests like PISA is similar to what is found in the table above for the Asian category. The fraction of PRC kids who perform in the top band is probably at least several times larger than the overall US fraction. (Asian vs White in the table above is about 6x, or 7x on the math portion.) Also, the fraction of college students who major in STEM is much larger in PRC than in the US.

This table was produced by German professor Gunnar Heinsohn, who analyzes geopolitics and human capital.

Note, I will censor racist comments.

Wednesday, September 28, 2022

The Future of Human Evolution -- excerpts from podcast interview with Brian Chau



1. The prospect of predicting cognitive ability from DNA, and the consequences. Why the main motivation has nothing to do with group differences. This segment begins at roughly 47 minutes. 

2. Anti-scientific resistance to research on the genetics of cognitive ability. My experience with the Jasons. Blank Slate-ism as a sacralized, cherished belief of social progressives. This segment begins at roughly 1 hour 7 minutes. 


1. Starts at roughly 47 minutes. 

Okay, let's just say hypothetically my billionaire friend is buddies with the CEO of 23andMe and let's say on the down low we collected some SAT scores of 1M or 2M people. I think there are about 10M people that have done 23andMe, let's suppose I manage to collect 1-2M scores for those people. I get them to opt in and agree to the study and da da da da and then Steve runs his algos and you get this nice predictor. 

But you’ve got to do it on the down low. Because if it leaks out that you're doing it, People are going to come for you. The New York Times is going to come for you, everybody's going to come for you. They're going to try to trash the reputation of 23andMe. They're going to trash the reputation of the billionaire. They're going to trash the reputation of the scientists who are involved in this. But suppose you get it done. And getting it done as you know very well is a simple run on AWS and you end up with this predictor which wow it's really complicated it depends on 20k SNPs in the genome ... 

For anybody with an ounce of intellectual integrity, they would look back at their copy of The Mismeasure of Man which has sat magisterially on their bookshelf since they were forced to buy it as a freshman at Harvard. They would say, “WOW! I guess I can just throw that in the trash right? I can just throw that in the trash.” 

But the set of people who have intellectual integrity and can process new information and then reformulate the opinion that they absorbed through social convention – i.e., that Gould is a good person and a good scientist and wise -- is tiny. The set of people who can actually do that is like 1% of the population. So you know maybe none of this matters, but in the long run it does matter. … 

Everything else about that hypothetical: the social scientists running the longitudinal study, getting the predictor in his grubby little hands and publishing the validation, but people trying to force you to studiously ignore the results, all that has actually already happened. We already have something which correlates ~0.4 with IQ. Everything else I said has already been done but it's just being studiously ignored by the right thinking people. 

 … 

Some people could misunderstand our discussion as being racist. I'm not saying that any of this has anything to do with group differences between ancestry groups. I'm just saying, e.g., within the white population of America, it is possible to predict from embryo DNA which of 2 brothers raised in the same family will be the smart one and which one will struggle in school. Which one will be the tall one and which one will be not so tall. 



2. Starts at roughly 1 hour 7 minutes. 

I've been in enough places where this kind of research is presented in seminar rooms and conferences and seen very negative attacks on the individuals presenting the results. 

I'll give you a very good example. There used to be a thing called the Jasons. During the cold war there was a group of super smart scientists called the Jasons. They were paid by the government to get together in the summers and think about technological issues that might be useful for defense and things like war fighting. … 

I had a meeting with the (current) Jasons. I was invited to a place near Stanford to address them about genetic engineering, genomics, and all this stuff. I thought okay these are serious scientists and I'll give them a very nice overview of the progress in this field. This anecdote takes place just a few years ago. 

One of the Jasons present is a biochemist but not an expert on genomics or machine learning. This biochemist asked me a few sharp questions which were easy to answer. But then at some point he just can't take it anymore and he grabs all his stuff and runs out of the room. ...

Sunday, August 14, 2022

Tweet Treats: AI in PRC, Semiconductors and the Russian War Machine, Wordcels are Midwits

Some recent tweets which might be of interest :-)

Saturday, July 16, 2022

Meritocracy and Political Leadership in China

Putting this tweet thread here for future reference. If you read this blog you may want to follow me on Twitter as I sometimes say things there that might be of interest.

Tuesday, May 03, 2022

How We Learned, Then Forgot, About Human Intelligence... And Witnessing the Live Breakdown of Academia (podcast interview with Cactus Chu)

This is a long interview I did recently with Cactus Chu, a math prodigy turned political theorist and podcaster. (Unfortunately I can't embed the podcast here.)


Timestamps: 
3:24 Interview Starts  
15:49 Cactus' Experience with High Math People 
19:49 High School Sports 
21:26 Comparison to Intelligence 
26:29 Is Lack of Understanding due to Denial or Ignorance? 
29:29 The Past and Present of Selection in Academia 
37:02 How Universities Look from the Inside 
44:19 Informal Networks Replacing Credentials 
48:37 Capture of Research Positions 
50:24 Progressivism as Demagoguery Against the Self-Made 
55:31 Innumeracy is Common 
1:06:53 Understanding Innumerate People 
1:13:53 Skill Alignment at Cactus' High School 
1:18:12 Free Speech in Academia 
1:21:00 You Shouldn't Fire Exceptional People 
1:23:03 The Anti-Excellence Progressives 
1:28:42 Rawls, Nozick, and Technology 
1:34:00 Freedom = Variance = Inequality 
1:37:58 Dating Apps 
1:41:27 Jumping Into Social Problems From a Technical Background 
1:41:50 Steve's High School Pranks 
1:46:43 996 and Cactus' High School 
1:50:26 The Vietnam War and Social Change 
1:53:07 Are Podcasts the Future? 
1:59:37 The Power of New Things 
2:02:56 The Birth of Twitter 
2:07:27 Selection Creates Quality 
2:10:21 Incentives of University Departments 
2:16:29 Woke Bureaucrats 
2:27:59 Building a New University 
2:30:42 What needs more order? 
2:31:56 What needs more chaos?

An automated (i.e., imperfect) transcript of our discussion.

Here's an excerpt from the podcast:

Monday, March 14, 2022

"The Pressure to Conform is Enormous": Steve Hsu on Affirmative Action, Assimilation and IQ Outliers (CSPI Podcast with Richard Hanania)

 

Another great conversation with Richard Hanania. 

Some rough timestamps: 
Begin: American society, growing up as child of immigrants 

18m: Russia-Ukraine conflict (eve of invasion), geopolitical implications (China, India, Germany, EU) 

38m: Affirmative Action, Harvard case at SCOTUS 

54m: Woke leftists at the university, destruction of meritocracy, STEM vs Social Justice advocacy, Sokal Hoax 

1h25m: Academic economics, 2008 credit crisis, Do economists test theories? 

1h33m: Maverick thinking, Agreeableness, Aspergers, Pressure to conform 

1h39m: Far-tail intelligence, Jeff Bezos and physics, progress in science and technology
Full transcript at Richard's substack.

Saturday, February 05, 2022

Annals of Psychometry: Wordcels and Shape Rotators


Fun with psychometrics! 

Did it all start with High V, Low M, a 2011 post about Stephen J. Gould?

A famous theoretical physicist once complained acerbically to me about someone's paper we were discussing:
It is nothing more than the calculus of words.
Yet there are people who have nothing more than the calculus of words with which to build their models of the world. See Bounded Cognition, and Oppenheimer:
Mathematics is "an immense enlargement of language, an ability to talk about things which in words would be simply inaccessible."

From A Song of Shapes and Words by Roon.
There are many verbally gifted writers and speakers that, when pressed to visualize some math problem in their mind's eye, must helplessly watch their normally high-octane intelligence sputter and fail. They often write or talk at a blistering clip, and can navigate complex mazes of abstractions — and yet, when it comes time to make contact with the real world or accomplish practical tasks, they may be helpless. They'll do great in English class, and terrible in Physics. They can be very fun to listen to due to their terrifying leaps in logic and the exceptional among them will be natural leaders. 
The wordcel moniker describes more than just one’s level of verbal skill: it’s also a socioeconomic classifier that refers to people whose verbal ability borders on self-sabotage (thus the “-cel”). Perhaps they’re driven mad by political rage, postmodernism, and disconnection from reality. It might refer to the priestly figures who work in the culture factories of the New York Times with their incomes and social prestige both precipitously declining only for the unperturbed masses on the internet to tell them in unison: “learn to code”! There’s even an implication that these folks are entirely rent-seekers (wrong, but directionally interesting). 
... 
The shape rotators have been a minor force until very recent history. Though they’ve produced a significant portion of human progress through feats of engineering excellence, they were rarely celebrated until the dawn of the Enlightenment, perhaps 500 years ago. While the long-lasting glory of the Roman aqueducts is renowned to this day, nobody knows the chief engineer behind the project (probably Marcus Vipsanius Agrippa, but who’s counting). Today their stock is climbing to the moon. The world’s richest (self-made) men are almost uniformly engineers, computer scientists, or physicists. Vast portions of society that in a prior age might have been organized by government bureaucrats or private sector shot-callers have been handed over to cybernetic self-organizing systems designed and run by mathematical wizards. We have been witness to the slow, and then rapid transfer of power from the smooth-talking Don Drapers of boardroom acclaim to the multi-armed bandits of Facebook Ads. 
It’s clear that these big tech CEOs are verbally gifted, but by affinity and by practice they are in the rotator camp. Elon continually attributes his success to studying physics in college. Zuck programmed the original iteration of Facebook himself. Larry & Sergei did an entire PhD in linear algebra based information retrieval, a platonic ideal of shape rotation. Of the ten largest companies in the world, several are driven by fundamental technical breakthroughs. Society at large seems to respect and fear the forces of technology more and more as its cultural and financial capital rises.

There is some conflation between Math ability and Spatial ability in this recent talk of Wordcels and Shape Rotators. Math and Spatial ability are positively correlated but are actually separate factors that emerge from PCA in psychometrics. Look carefully at the arrows in the figure below -- if you can't read the figure you might be a wordcel ;-)

Note also that in the SMPY/SVPY data physicists dominated the wordcels even in their own verbal domain. This is also confirmed here.


See post from 2016 reproduced below, especially point #3.
3. There are systematic differences in cognitive abilities and profiles in different fields (business, medicine, engineering, physics, etc.)
This figure displays the math, verbal and spatial scores of gifted children tested at age 12, and their eventual college majors and career choices. This group is cohort 2 of the SMPY/SVPY study: each child scored better than 99.5 percentile on at least one of the M-V sections of the SAT.





Scores are normalized in units of SDs, within this cohort of gifted children. (So above and below average are defined with respect to the gifted population of >99th percentile kids, not relative to the general population.) The vertical axis is V, the horizontal axis is M, and the length of the arrow reflects spatial ability: pointing to the right means above the group average, to the left means below average; note the arrow for business majors should be twice as long as indicated but there was not enough space on the diagram. The spatial score is obviously correlated with the M score. More data here.


SMPY at 50: Research Associate position (2016)

I'm posting the job ad below for David Lubinski. The Study of Mathematically Precocious Youth (SMPY) is the most systematic long term study of individuals of high cognitive ability since the Terman Study.

SMPY helps to establish a number of important facts about individuals of high ability:

1. We can (at least crudely) differentiate between individuals at the 99th, 99.9th and 99.99th percentiles. Exceptional talent can be identified through testing, even at age 13.

2. Probability of significant accomplishment, such as STEM PhD, patents awarded, tenure at leading research university, exceptional income, etc. continues to rise as ability level increases, even within the top 1%.

3. There are systematic differences in cognitive abilities and profiles in different fields (business, medicine, engineering, physics, etc.)

4. Men and women of exceptional ability differ in life aspirations and preferences.

No one can claim to understand high level human capital, technological innovation, scientific progress, or exceptional achievement without first familiarizing themselves with these results.

Needless to say, I think this Research Associate position will entail important and fascinating work.
Research Associate:

The Study of Mathematically Precocious Youth (SMPY) seeks a full-time post-doctoral Research Associate for study oversight, conducting research, writing articles, laboratory management, and statistical analyses using the vast SMPY data base. SMPY is a four-decade longitudinal study consisting of 5 cohorts and over 5,000 intellectually talented participants. One chief responsibility of this position will be to manage laboratory details associated with launching an age-50 follow-up of two of SMPY’s most exceptional cohorts: a cohort of 500 profoundly gifted participants initially identified by age 13 in the early 1980s, and a second cohort of over 700 top STEM graduate students identified and psychologically profiled in 1992 as first- and second-year graduate students. Candidates with interests in assessing individual differences, talent development, and particularly strong statistical-technical skills are preferred. Send vitae, cover letter stating interests, (pre)reprints, and three letters of recommendation to: Dean Camilla P. Benbow, Department of Psychology & Human Development, 0552 Peabody College, Vanderbilt University, Nashville, TN, 37203. The position will remain open until a qualified applicant is selected. For additional information, please contact either co-director: Camilla P. Benbow, camilla.benbow@vanderbilt.edu, or David Lubinski, david.lubinski@vanderbilt.edu.

http://www.vanderbilt.edu/Peabody/SMPY/. Vanderbilt University is an Equal Opportunity/Affirmative Action Employer.

We are aiming for a June 30th start date but that’s flexible.
Some relevant figures based on SMPY results of Lubinski, Benbow, and collaborators. See links above for more discussion of the data displayed.











Monday, January 10, 2022

Recent Papers on Socio-Economic Status and Student Achievement: Marks and O'Connell

I received the message below from Michael O'Connell, University College Dublin, and Gary Marks, University of Melbourne. 

See also this recent post: Social and Educational Mobility: Denmark vs USA (James Heckman), and links therein. 
Dear Scholar, 
 
There is a widely-held perception that many of life’s key outcomes are fundamentally driven by people’s socio-economic status (SES). More specifically, there is a view that children’s educational attainment is largely a by-product of their familial SES. As a consequence of this pervasive paradigm, much of the energy in seeking to ameliorate or resolve poor educational attainment is based around trying to use SES as a social lever. 

However, in the six papers listed below, published between 2019-2022, evidence has been gathered demonstrating that SES is only very modestly correlated with educational attainment. Furthermore, once a child’s cognitive ability is taken into account, even the modest link between SES and attainment diminishes to slight influence. This is true of datasets drawn from international groups of young people, as well as those from the US, UK, or Ireland. Future attempts to aid and study young people experiencing difficulty with educational attainment should be built on an awareness of the limited role of SES. 

Gary N Marks   Michael O’Connell 


1. O’Connell, M. and Marks, G.N. (2022) 
Cognitive ability and conscientiousness are more important than SES for educational attainment: An analysis of the UK Millennium Cohort Study
Personality and Individual Differences, 188 
https://doi.org/10.1016/j.paid.2021.111471 
Highlights Antecedents of educational attainment of great interest Dominant paradigm focuses on SES of children. Cognitive ability and conscientiousness have stronger record in research findings. Using new UK MCS longitudinal survey data, GCSE state exam performance assessed Cognitive ability and conscientiousness explained far more than SES measures 

 

2. Marks, G. N. (2021) 
Is the relationship between socioeconomic status (SES) and student achievement causal? Considering student and parent abilities
Educational Research and Evaluation, 10.1080/13803611.2021.1968442: 1-24. 
Abstract Most studies on the relationship between students’ socioeconomic status (SES) and student achievement assume that its effects are sizable and causal. A large variety of theoretical explanations have been proposed. However, the SES–achievement association may reflect, to some extent, the inter-relationships of parents’ abilities, SES, children’s abilities, and student achievement. The purpose of this study is to quantify the role of SES vis-à-vis child and parents’ abilities, and prior achievement. Analyses of a covariance matrix that includes supplementary correlations for fathers and mothers’ abilities derived from the literature indicate that more than half of the SES–achievement association can be accounted for by parents’ abilities. SES coefficients decline further with the addition of child’s abilities. With the addition of prior achievement, the SES coefficients are trivial implying that SES has little or no contemporaneous effects. These findings are not compatible with standard theoretical explanations for SES inequalities in achievement. 

 

3. Marks, G. N. and O’Connell, M. (2021) 
No Evidence for Cumulating Socioeconomic Advantage. Ability Explains Increasing SES Effects with Age on Children’s Domain Test scores 
Intelligence, 88    
https://doi.org/10.1016/j.intell.2021.101582 
Highlights Data analysed for five domains for children of the NLSY79 mothers study. SES effects increase for only some domains and not substantially. No increase in SES effects when considering mother's or children's prior ability. Effects of child's prior ability on test scores increase substantially with age. SES effects are small net of mother's ability. 
 
4. Marks, G. N. and O'Connell, M. (2021) 
Inadequacies in the SES–Achievement model: Evidence from PISA and other studies 
Review of Education, 9(3): e3293. 
https://doi.org/10.1002/rev3.3293 
Abstract Students’ socioeconomic status (SES) is central to much research and policy deliberation on educational inequalities. However, the SES model is under severe stress for several reasons. SES is an ill-defined concept, unlike parental education or family income. SES measures are frequently based on proxy reports from students; these are generally unreliable, sometimes endogenous to student achievement, only low to moderately intercorrelated, and exhibit low comparability across countries and over time. There are many explanations for SES inequalities in education, none of which achieves consensus among research and policy communities. SES has only moderate effects on student achievement, and its effects are especially weak when considering prior achievement, an important and relevant predictor. SES effects are substantially reduced when considering parent ability, which is causally prior to family SES. The alternative cognitive ability/genetic transmission model has far greater explanatory power; it provides logical and compelling explanations for a wide range of empirical findings from student achievement studies. The inadequacies of the SES model are hindering knowledge accumulation about student performance and the development of successful policies. 
 
5. O'Connell, M. and Marks, G. N. (2021) 
Are the effects of intelligence on student achievement and well-being largely functions of family income and social class? Evidence from a longitudinal study of Irish adolescents
Intelligence, 84: 101511. 10.1016/j.intell.2020.101511 
Highlights Power of cognitive ability and social class contrasted. Large representative sample from longitudinal study, waves 1–3, of 6216 children Outcomes were attainments, difficulties and relationships. Cognitive ability explained large amounts of variance. Social background only minor effects 
 
6. O'Connell, M. (2019) 
Is the impact of SES on educational performance overestimated? Evidence from the PISA survey 
Intelligence, 75: 41-47 
https://doi.org/10.1016/j.intell.2019.04.005 
Highlights Policy-makers overly attribute differences in educational performance to SES. PISA survey used to assess roles of parental education and household income. Combining them concealed differences in outcomes between rich and poor countries. Household income important in poor countries, parental education in rich countries.

Tuesday, September 07, 2021

Kathryn Paige Harden Profile in The New Yorker (Behavior Genetics)

This is a good profile of behavior geneticist Paige Harden (UT Austin professor of psychology, former student of Eric Turkheimer), with a balanced discussion of polygenic prediction of cognitive traits and the culture war context in which it (unfortunately) exists.
Can Progressives Be Convinced That Genetics Matters? 
The behavior geneticist Kathryn Paige Harden is waging a two-front campaign: on her left are those who assume that genes are irrelevant, on her right those who insist that they’re everything. 
Gideon Lewis-Kraus
Gideon Lewis-Kraus is a talented writer who also wrote a very nice article on the NYTimes / Slate Star Codex hysteria last summer.

Some references related to the New Yorker profile:
1. The paper Harden was attacked for sharing while a visiting scholar at the Russell Sage Foundation: Game Over: Genomic Prediction of Social Mobility 

2. Harden's paper on polygenic scores and mathematics progression in high school: Genomic prediction of student flow through high school math curriculum 

3. Vox article; Turkheimer and Harden drawn into debate including Charles Murray and Sam Harris: Scientific Consensus on Cognitive Ability?

A recent talk by Harden, based on her forthcoming book The Genetic Lottery: Why DNA Matters for Social Equality



Regarding polygenic prediction of complex traits 

I first met Eric Turkheimer in person (we had corresponded online prior to that) at the Behavior Genetics Association annual meeting in 2012, which was back to back with the International Conference on Quantitative Genetics, both held in Edinburgh that year (photos and slides [1] [2] [3]). I was completely new to the field but they allowed me to give a keynote presentation (if memory serves, together with Peter Visscher). Harden may have been at the meeting but I don't recall whether we met. 

At the time, people were still doing underpowered candidate gene studies (there were many talks on this at BGA although fewer at ICQG) and struggling to understand GCTA (Visscher group's work showing one can estimate heritability from modestly large GWAS datasets, results consistent with earlier twins and adoption work). Consequently a theoretical physicist talking about genomic prediction using AI/ML and a million genomes seemed like an alien time traveler from the future. Indeed, I was.

My talk is largely summarized here:
On the genetic architecture of intelligence and other quantitative traits 
https://arxiv.org/abs/1408.3421 
How do genes affect cognitive ability or other human quantitative traits such as height or disease risk? Progress on this challenging question is likely to be significant in the near future. I begin with a brief review of psychometric measurements of intelligence, introducing the idea of a "general factor" or g score. The main results concern the stability, validity (predictive power), and heritability of adult g. The largest component of genetic variance for both height and intelligence is additive (linear), leading to important simplifications in predictive modeling and statistical estimation. Due mainly to the rapidly decreasing cost of genotyping, it is possible that within the coming decade researchers will identify loci which account for a significant fraction of total g variation. In the case of height analogous efforts are well under way. I describe some unpublished results concerning the genetic architecture of height and cognitive ability, which suggest that roughly 10k moderately rare causal variants of mostly negative effect are responsible for normal population variation. Using results from Compressed Sensing (L1-penalized regression), I estimate the statistical power required to characterize both linear and nonlinear models for quantitative traits. The main unknown parameter s (sparsity) is the number of loci which account for the bulk of the genetic variation. The required sample size is of order 100s, or roughly a million in the case of cognitive ability.
The predictions in my 2012 BGA talk and in the 2014 review article above have mostly been validated. Research advances often pass through the following phases of reaction from the scientific community:
1. It's wrong ("genes don't affect intelligence! anyway too complex to figure out... we hope")
2. It's trivial ("ofc with lots of data you can do anything... knew it all along")
3. I did it first ("please cite my important paper on this")
Or, as sometimes attributed to Gandhi: "First they ignore you, then they laugh at you, then they fight you, then you win.”



Technical note

In 2014 I estimated that ~1 million genotype | phenotype pairs would be enough to capture most of the common SNP heritability for height and cognitive ability. This was accomplished for height in 2017. However, the sample size of well-phenotyped individuals is much smaller for cognitive ability, even in 2021, than for height in 2017. For example, in UK Biobank the cognitive test is very brief (~5 minutes IIRC, a dozen or so questions), but it has not even been administered to the full cohort as yet. In the Educational Attainment studies the phenotype EA is only moderately correlated (~0.3 ?) or so with actual cognitive ability.

Hence, although the most recent EA4 results use 3 million individuals [1], and produce a predictor which correlates ~0.4 with actual EA, the statistical power available is still less than what I predicted would be required to train a really good cognitive ability predictor.

In our 2017 height paper, which also briefly discussed bone density and cognitive ability prediction, we built a cognitve ability predictor roughly as powerful as EA3 using only ~100k individuals with the noisy UKB test data. So I remain confident that  ~million individuals with good cognitive scores (e.g., SAT, AFQT, full IQ test) would deliver results far beyond what we currently have available. We also found that our predictor, built using actual (albeit noisy) cognitive scores exhibits less power reduction in within-family (sibling) analyses compared to EA. So there is evidence that (no surprise) EA is more influenced by environmental factors, including so-called genetic nurture effects, than is cognitive ability.

A predictor which captures most of the common SNP heritability for cognitive ability might correlate ~0.5 or 0.6 with actual ability. Applications of this predictor in, e.g., studies of social mobility or educational success or even longevity using existing datasets would be extremely dramatic.

Sunday, April 18, 2021

Francois Chollet - Intelligence and Generalization, Psychometrics for Robots (AI/ML)

 

If you have thought a lot about AI and deep learning you may find much of this familiar. Nevertheless I enjoyed the discussion. Apparently Chollet's views (below) are controversial in some AI/ML communities but I do not understand why. 

Chollet's Abstraction and Reasoning Corpus (ARC) = Raven's Matrices for AIs :-)
Show Notes: 
...Francois has a clarity of thought that I've never seen in any other human being! He has extremely interesting views on intelligence as generalisation, abstraction and an information conversation ratio. He wrote on the measure of intelligence at the end of 2019 and it had a huge impact on my thinking. He thinks that NNs can only model continuous problems, which have a smooth learnable manifold and that many "type 2" problems which involve reasoning and/or planning are not suitable for NNs. He thinks that many problems have type 1 and type 2 enmeshed together. He thinks that the future of AI must include program synthesis to allow us to generalise broadly from a few examples, but the search could be guided by neural networks because the search space is interpolative to some extent. 
Tim Intro [00:00:00​]
Manifold hypothesis and interpolation [00:06:15​]
Yann LeCun skit [00:07:58​]
Discrete vs continuous [00:11:12​]
NNs are not turing machines [00:14:18​]
Main show kick-off [00:16:19​]
DNN models are locally sensitive hash tables and only efficiently encode some kinds of data well [00:18:17​]
Why do natural data have manifolds? [00:22:11​]
Finite NNs are not "turing complete" [00:25:44​]
The dichotomy of continuous vs discrete problems, and abusing DL to perform the former [00:27:07​]
Reality really annoys a lot of people, and ...GPT-3 [00:35:55​]
There are type one problems and type 2 problems, but...they are enmeshed [00:39:14​]
Chollet's definition of intelligence and how to construct analogy [00:41:45​]
How are we going to combine type 1 and type 2 programs? [00:47:28​]
Will topological analogies be robust and escape the curse of brittleness? [00:52:04​]
Is type 1 and 2 two different physical systems? Is there a continuum? [00:54:26​]
Building blocks and the ARC Challenge [00:59:05​]
Solve ARC == intelligent? [01:01:31​]
Measure of intelligence formalism -- it's a whitebox method [01:03:50​]
Generalization difficulty [01:10:04​]
Lets create a marketplace of generated intelligent ARC agents! [01:11:54​]
Mapping ARC to psychometrics [01:16:01​]
Keras [01:16:45​]
New backends for Keras? JAX? [01:20:38​]
Intelligence Explosion [01:25:07​]
Bottlenecks in large organizations [01:34:29​]
Summing up the intelligence explosion [01:36:11​]
Post-show debrief [01:40:45​]
This is Chollet's paper which is the focus of much of the discussion.
On the Measure of Intelligence 
François Chollet   
https://arxiv.org/abs/1911.01547 
To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans. Over the past hundred years, there has been an abundance of attempts to define and measure intelligence, across both the fields of psychology and AI. We summarize and critically assess these definitions and evaluation approaches, while making apparent the two historical conceptions of intelligence that have implicitly guided them. We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at specific tasks such as board games and video games. We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to "buy" arbitrary levels of skills for a system, in a way that masks the system's own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. Using this definition, we propose a set of guidelines for what a general AI benchmark should look like. Finally, we present a benchmark closely following these guidelines, the Abstraction and Reasoning Corpus (ARC), built upon an explicit set of priors designed to be as close as possible to innate human priors. We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.
Notes on the paper by Robert Lange (TU-Berlin), including illustrations like the ones below.





Sunday, March 21, 2021

The Contribution of Cognitive and Noncognitive Skills to Intergenerational Social Mobility (McGue et al. 2020)

If you have the slightest pretension to expertise concerning social mobility, meritocracy, inequality, genetics, psychology, economics, education, history, or any related subjects, I urge you to carefully study this paper.
The Contribution of Cognitive and Noncognitive Skills to Intergenerational Social Mobility  
(Psychological Science https://doi.org/10.1177/0956797620924677)
Matt McGue, Emily A. Willoughby, Aldo Rustichini, Wendy Johnson, William G. Iacono, James J. Lee 
We investigated intergenerational educational and occupational mobility in a sample of 2,594 adult offspring and 2,530 of their parents. Participants completed assessments of general cognitive ability and five noncognitive factors related to social achievement; 88% were also genotyped, allowing computation of educational-attainment polygenic scores. Most offspring were socially mobile. Offspring who scored at least 1 standard deviation higher than their parents on both cognitive and noncognitive measures rarely moved down and frequently moved up. Polygenic scores were also associated with social mobility. Inheritance of a favorable subset of parent alleles was associated with moving up, and inheritance of an unfavorable subset was associated with moving down. Parents’ education did not moderate the association of offspring’s skill with mobility, suggesting that low-skilled offspring from advantaged homes were not protected from downward mobility. These data suggest that cognitive and noncognitive skills as well as genetic factors contribute to the reordering of social standing that takes place across generations.
From the paper:
We believe that a reasonable explanation of our findings is that the degree to which individuals are more or less skilled than their parents contributes to their upward or downward mobility. Behavioral genetic and genomic research has established the heritability of social achievements (Conley, 2016) as well as the skills thought to underlie them (Bouchard & McGue, 2003). Nonetheless, these associations may be due to passive gene–environment correlation, whereby high-achieving parents both transmit genes and provide a rearing environment that promotes their children’s social success (Scarr & McCartney, 1983). Our within-family design controlled for passive gene–environment correlation effects. Although offspring inherit all of their genes from their parents, they inherit a random subset of parental alleles because of meiotic segregation. Consequently, some offspring inherit a favorable subset of their parents’ alleles, whereas others inherit a less favorable subset. We found, as did previous researchers (Belsky et al., 2018), that the inheritance of a favorable subset of alleles was associated with an increased likelihood of upward mobility... 
...In summary, our analysis of intergenerational social mobility in a sample of 2,594 offspring from 1,321 families found that (a) most individuals were educationally and occupationally mobile, (b) mobility was predicted by offspring–parent differences in skills and genetic endowment, and (c) the relationship of offspring skills with social mobility did not vary significantly by parent social background. In an era in which there is legitimate concern over social stagnation, our findings are noteworthy in identifying the circumstances when parents’ educational and occupational success is not reproduced across generations.

See also Game Over: Genomic Prediction of Social Mobility (PNAS July 9, 2018: 201801238). Both papers provide out of sample validation of polygenic predictors for cognitive ability, specifically of the relationship to intergenerational social mobility.


Thursday, May 21, 2020

University of California to end use of SAT and ACT

University of California Will End Use of SAT and ACT in Admissions (NYT)

This decision by the UC Regents (most of whom are political appointees) is counter to the recommendation of the faculty task force recently assigned to study standardized testing in admissions. It is obvious to anyone who looks at the graphs below that SAT/ACT have significant validity (technical term used in psychometrics) in predicting college performance for all ethnic groups.


See Report of the University of California Academic Council Standardized Testing Task Force for more.
... SAT and HSGPA are stronger predictors than family income or race. Within each of the family income or ethnicity categories there is substantial variation in SAT and HSGPA, with corresponding differences in student success. See bottom figure and combined model R^2 in second figure below; R^2 varies very little across family income and ethnic categories. ...

Test Preparation and SAT scores: "...combined effect of coaching on the SAT I is between 21 and 34 points. Similarly, extensive meta-analyses conducted by Betsy Jane Becker in 1990 and by Nan Laird in 1983 found that the typical effect of commercial preparatory courses on the SAT was in the range of 9-25 points on the verbal section, and 15-25 points on the math section."

Tuesday, February 04, 2020

Report of the University of California Academic Council Standardized Testing Task Force

The figures below are from the recently completed Report of the University of California Academic Council Standardized Testing Task Force. Note the large sample sizes.

Some remarks:

1. SAT and High School GPA (HSGPA) are both useful (and somewhat independent) predictors of college success. In terms of variance accounted for, we have the inequality:

SAT + HSGPA  >  SAT  >  HSGPA

There are some small deviations from this pattern, but it seems to hold overall. I believe that GPA has a relatively larger loading on conscientiousness (work ethic) than cognitive ability, with SAT the other way around. By combining the two we get more information than from either alone.

2. SAT and HSGPA are stronger predictors than family income or race. Within each of the family income or ethnicity categories there is substantial variation in SAT and HSGPA, with corresponding differences in student success. See bottom figure and combined model R^2 in second figure below; R^2 varies very little across family income and ethnic categories.







There is not much new here. In graduate admissions the undergraduate GPA and the GRE general + subject tests play a role similar to HSGPA and SAT. See GRE and SAT Validity.

See Correlation and Variance to understand better what the R^2 numbers above mean. R^2 ~ 0.26 means the correlation between predictor and outcome variable (e.g., freshman GPA) is R ~ 0.5 or so.

Test Preparation and SAT scores: "...combined effect of coaching on the SAT I is between 21 and 34 points. Similarly, extensive meta-analyses conducted by Betsy Jane Becker in 1990 and by Nan Laird in 1983 found that the typical effect of commercial preparatory courses on the SAT was in the range of 9-25 points on the verbal section, and 15-25 points on the math section."

Blog Archive

Labels