Saturday, May 17, 2014

Intervention and Replication

This article describes surprisingly effective interventions that improve college success rates for low SES/SAT students. Let's hope they can be broadly replicated.
NYTimes: ... Laude was hopeful that the small classes would make a difference, but he recognized that small classes alone wouldn’t overcome that 200-point SAT gap. “We weren’t na├»ve enough to think they were just going to show up and start getting A’s, unless we overwhelmed them with the kind of support that would make it possible for them to be successful,” he said. So he supplemented his lectures with a variety of strategies: He offered TIP students two hours each week of extra instruction; he assigned them advisers who kept in close contact with them and intervened if the students ran into trouble or fell behind; he found upperclassmen to work with the TIP students one on one, as peer mentors. And he did everything he could, both in his lectures and outside the classroom, to convey to the TIP students a new sense of identity: They weren’t subpar students who needed help; they were part of a community of high-achieving scholars.

Even Laude was surprised by how effectively TIP worked. “When I started giving them the tests, they got the same grades as the larger section,” he said. “And when the course was over, this group of students who were 200 points lower on the SAT had exactly the same grades as the students in the larger section.” The impact went beyond Chemistry 301. This cohort of students who, statistically, were on track to fail returned for their sophomore year at rates above average for the university as a whole, and three years later they had graduation rates that were also above the U.T. average.

... Laude wanted something that would help him predict, for any given incoming freshman, how likely he or she would be to graduate in four years. The Institutional Research team analyzed the performance of tens of thousands of recent U.T. students, and from that analysis they produced a tool they called the Dashboard — an algorithm, in spreadsheet form, that would consider 14 variables, from an incoming student’s family income to his SAT score to his class rank to his parents’ educational background, and then immediately spit out a probability, to the second decimal place, of how likely he was to graduate in four years.

In the spring of 2013, Laude and his staff sat down with the Dashboard to analyze the 7,200 high-school seniors who had just been admitted to the class of 2017. When they ran the students’ data, the Dashboard indicated that 1,200 of them — including Vanessa Brewer — had less than a 40 percent chance of graduation on time. Those were the kids Laude decided to target. He assigned them each to one or more newly created or expanded interventions. The heart of the project is a portfolio of “student success programs,” each one tailored, to a certain extent, for a different college at U.T. — natural sciences, liberal arts, engineering — but all of them following the basic TIP model Laude dreamed up 15 years ago: small classes, peer mentoring, extra tutoring help, engaged faculty advisers and community-building exercises.

... Students were randomly sorted into four categories. A “belonging” treatment group read messages from current students explaining that they felt alone and excluded when they arrived on campus, but then realized that everyone felt that way and eventually began to feel at home. A “mind-set” treatment group read an article about the malleability of the brain and how practice makes it grow new connections, and then read messages from current students stating that when they arrived at U.T., they worried about not being smart enough, but then learned that when they studied they grew smarter. A combination treatment group received a hybrid of the belonging and mind-set presentations. And finally, a control group read fairly banal reflections from current students stating that they were surprised by Austin’s culture and weather when they first arrived, but eventually they got used to them. Students in each group were asked, after clicking through a series of a dozen or so web pages, to write their own reflections on what they’d read in order to help future students. The whole intervention took between 25 and 45 minutes for students to complete, and more than 90 percent of the incoming class completed it.

... In January 2013, when Yeager analyzed the first-semester data, he saw the advantaged students’ results were exactly the same as they were every year. No matter which message they saw in the pre-orientation presentation, 90 percent of that group was on track. Similarly, the disadvantaged students in the control group, who saw the bland message about adjusting to Austin’s culture and weather, did the same as disadvantaged students usually did: 82 percent were on track. But the disadvantaged students who had experienced the belonging and mind-set messages did significantly better: 86 percent of them had completed 12 credits or more by Christmas. They had cut the gap between themselves and the advantaged students in half.
One problem with this type of treatment or intervention is that the selected group should not be stigmatized, or even aware of why they were selected for the special treatment. As awareness grows (e.g., through this high profile article in the NYTimes magazine), it becomes harder and harder to camouflage the criteria used to select students for programs like TIP. Note, in the last intervention described above it appears the entire freshman class (~ 7k students) at UT Austin was used, so sample size was fairly large.


JayMan said...

Interesting. Sounds way too good to be true, though.

ben_g said...

"it becomes harder and harder to camouflage the criteria used to select students for programs like TIP"

Online education can help with this. When the class doesn't need to follow the same pace, the students with atypical learning curves can prosper rather than be dragged along or alternatively bored. And since it's all online, there's no visible stigma with making your way through the material more slowly.

My experience with school was a combination of being a troubled kid (subpar grades + rule breaking) and being bored. The school system rewards conforming and average behavior, at the expense of troubled and/or gifted kids.

Wish Khan Academy was around when I was in school: . Turns out that in the real world, actually understanding things and coming up with your own ideas proves way more valuable than conforming to the rigid criteria of those in power.

larryholmes said...

"Intelligence" is a noun. There is no English verb meaning "to behave as an intelligent person behaves". As a result, "intelligence" is thought of as something which is "just there".

Charlie, of Flowers for Algernon, had a lot to learn if you remember. Man is a social animal like no other.

Lemongrab said...

When i was a post grad i did the statistical analysis for an internal university report on dropout rates and we found that for the bottom 1/3 of students parental SES was more important (better predictor) than SAT score. The reasons for this may be complicated, more complicated than the NYT article implies, but it did suggest to us that their was a problem in the way the university responded to students at risk of dropping out. Contrary to the NYT article (and general claims) about brain plasticity it does seem difficult to create long term improvements in IQ. On the other hand perhaps we overestimate the general mental ability required to complete a given course/task. Their seems to be a variety of research (James Heckman, or Raj Chetty & class size effects and again here) that suggests that outcomes at a given SAT/IQ level are not fixed/can be improved. All these studies seem to demonstrate that some part of the gap between normal and disadvantaged students can be closed by targeted interventions. Obviously, if correct, this would complicate the picture of a simple relationship between IQ and social outcomes and it would also be great for society.

DK said...

Two data points (1200-1399 and 1400-1600) were replaced with a single one. Why? To keep the gap visually about the same?

Emil Kirkegaard said...

This might also be of interest to you:

johnholmes14inches said...

Of course, you could...

just like the stories about the genes for this or that psychological trait.

James Hedman said...

"Alpha children wear grey They work much harder than we do, because they're so frightfully clever. I'm really awfuly glad I'm a Beta, because I don't work so hard. And then we are much better than the Gammas and Deltas. Gammas are stupid. They all wear green, and Delta children wear khaki. Oh no, I don't want to play with Delta children. And Epsilons are still worse. They're too stupid to be able …"

5371 said...

Like most results in psychology, these are doubtless not replicable.

ben_g said...

Nice video. @steve hsu might wanna fix up the ugly way these videos display though

Blog Archive