Intervention and Replication

This article describes surprisingly effective interventions that improve college success rates for low SES/SAT students. Let's hope they can be broadly replicated.

NYTimes: ... Laude was hopeful that the small classes would make a difference, but he recognized that small classes alone wouldn’t overcome that 200-point SAT gap. “We weren’t naïve enough to think they were just going to show up and start getting A’s, unless we overwhelmed them with the kind of support that would make it possible for them to be successful,” he said. So he supplemented his lectures with a variety of strategies: He offered TIP students two hours each week of extra instruction; he assigned them advisers who kept in close contact with them and intervened if the students ran into trouble or fell behind; he found upperclassmen to work with the TIP students one on one, as peer mentors. And he did everything he could, both in his lectures and outside the classroom, to convey to the TIP students a new sense of identity: They weren’t subpar students who needed help; they were part of a community of high-achieving scholars.

Even Laude was surprised by how effectively TIP worked. “When I started giving them the tests, they got the same grades as the larger section,” he said. “And when the course was over, this group of students who were 200 points lower on the SAT had exactly the same grades as the students in the larger section.” The impact went beyond Chemistry 301. This cohort of students who, statistically, were on track to fail returned for their sophomore year at rates above average for the university as a whole, and three years later they had graduation rates that were also above the U.T. average.

... Laude wanted something that would help him predict, for any given incoming freshman, how likely he or she would be to graduate in four years. The Institutional Research team analyzed the performance of tens of thousands of recent U.T. students, and from that analysis they produced a tool they called the Dashboard — an algorithm, in spreadsheet form, that would consider 14 variables, from an incoming student’s family income to his SAT score to his class rank to his parents’ educational background, and then immediately spit out a probability, to the second decimal place, of how likely he was to graduate in four years.

In the spring of 2013, Laude and his staff sat down with the Dashboard to analyze the 7,200 high-school seniors who had just been admitted to the class of 2017. When they ran the students’ data, the Dashboard indicated that 1,200 of them — including Vanessa Brewer — had less than a 40 percent chance of graduation on time. Those were the kids Laude decided to target. He assigned them each to one or more newly created or expanded interventions. The heart of the project is a portfolio of “student success programs,” each one tailored, to a certain extent, for a different college at U.T. — natural sciences, liberal arts, engineering — but all of them following the basic TIP model Laude dreamed up 15 years ago: small classes, peer mentoring, extra tutoring help, engaged faculty advisers and community-building exercises.

... Students were randomly sorted into four categories. A “belonging” treatment group read messages from current students explaining that they felt alone and excluded when they arrived on campus, but then realized that everyone felt that way and eventually began to feel at home. A “mind-set” treatment group read an article about the malleability of the brain and how practice makes it grow new connections, and then read messages from current students stating that when they arrived at U.T., they worried about not being smart enough, but then learned that when they studied they grew smarter. A combination treatment group received a hybrid of the belonging and mind-set presentations. And finally, a control group read fairly banal reflections from current students stating that they were surprised by Austin’s culture and weather when they first arrived, but eventually they got used to them. Students in each group were asked, after clicking through a series of a dozen or so web pages, to write their own reflections on what they’d read in order to help future students. The whole intervention took between 25 and 45 minutes for students to complete, and more than 90 percent of the incoming class completed it.

... In January 2013, when Yeager analyzed the first-semester data, he saw the advantaged students’ results were exactly the same as they were every year. No matter which message they saw in the pre-orientation presentation, 90 percent of that group was on track. Similarly, the disadvantaged students in the control group, who saw the bland message about adjusting to Austin’s culture and weather, did the same as disadvantaged students usually did: 82 percent were on track. But the disadvantaged students who had experienced the belonging and mind-set messages did significantly better: 86 percent of them had completed 12 credits or more by Christmas. They had cut the gap between themselves and the advantaged students in half.

One problem with this type of treatment or intervention is that the selected group should not be stigmatized, or even aware of why they were selected for the special treatment. As awareness grows (e.g., through this high profile article in the NYTimes magazine), it becomes harder and harder to camouflage the criteria used to select students for programs like TIP. Note, in the last intervention described above it appears the entire freshman class (~ 7k students) at UT Austin was used, so sample size was fairly large.