Tuesday, December 16, 2008

Teaching effectiveness

The two figures below (click for larger versions) are taken from the Brookings report by Gordon, Kane and Staiger: Identifying Effective Teachers Using Performance on the Job. The report has received a lot of attention recently thanks to Malcolm Gladwell's New Yorker article. Both are worth a look if you are interested in education. The top figure shows that certification has no impact on teaching effectiveness. The second shows that effectiveness measured in the years 1 and 2 is predictive of effectiveness in the subsequent year. In this case effectiveness is defined by the average change in percentile ranking of students in the teacher's class. Good teachers help their students to improve their mastery, hence percentile ranking, relative to the average student studying the same material.

It's obvious to me that there is gigantic variation in effectiveness among teachers. Gladwell emphasizes how difficult it is to evaluate teaching capability in initial hiring, and how the single most important impact on overall school effectiveness is due to individual teachers (he also makes the analogy to scouting college QBs for pro football -- it's very hard to predict NFL performance based on college performance). The Brookings paper has many policy suggestions, but the basic idea is that if we were disciplined and data-driven we could easily determine which teachers are good and which ones are not.

New Yorker: ...One of the most important tools in contemporary educational research is “value added” analysis. It uses standardized test scores to look at how much the academic performance of students in a given teacher’s classroom changes between the beginning and the end of the school year. Suppose that Mrs. Brown and Mr. Smith both teach a classroom of third graders who score at the fiftieth percentile on math and reading tests on the first day of school, in September. When the students are retested, in June, Mrs. Brown’s class scores at the seventieth percentile, while Mr. Smith’s students have fallen to the fortieth percentile. That change in the students’ rankings, value-added theory says, is a meaningful indicator of how much more effective Mrs. Brown is as a teacher than Mr. Smith.

It’s only a crude measure, of course. A teacher is not solely responsible for how much is learned in a classroom, and not everything of value that a teacher imparts to his or her students can be captured on a standardized test.

Nonetheless, if you follow Brown and Smith for three or four years, their effect on their students’ test scores starts to become predictable: with enough data, it is possible to identify who the very good teachers are and who the very poor teachers are. What’s more—and this is the finding that has galvanized the educational world—the difference between good teachers and poor teachers turns out to be vast.

Eric Hanushek, an economist at Stanford, estimates that the students of a very bad teacher will learn, on average, half a year’s worth of material in one school year. The students in the class of a very good teacher will learn a year and a half’s worth of material. That difference amounts to a year’s worth of learning in a single year. Teacher effects dwarf school effects: your child is actually better off in a “bad” school with an excellent teacher than in an excellent school with a bad teacher. Teacher effects are also much stronger than class-size effects. You’d have to cut the average class almost in half to get the same boost that you’d get if you switched from an average teacher to a teacher in the eighty-fifth percentile. And remember that a good teacher costs as much as an average one, whereas halving class size would require that you build twice as many classrooms and hire twice as many teachers.

Hanushek recently did a back-of-the-envelope calculation about what even a rudimentary focus on teacher quality could mean for the United States. If you rank the countries of the world in terms of the academic performance of their schoolchildren, the U.S. is just below average, half a standard deviation below a clump of relatively high-performing countries like Canada and Belgium. According to Hanushek, the U.S. could close that gap simply by replacing the bottom six per cent to ten per cent of public-school teachers with teachers of average quality. After years of worrying about issues like school funding levels, class size, and curriculum design, many reformers have come to the conclusion that nothing matters more than finding people with the potential to be great teachers. But there’s a hitch: no one knows what a person with the potential to be a great teacher looks like. The school system has a quarterback problem.

In my experience as a university professor I find that most colleagues think of themselves as above-average teachers, even when they are not. Essentially no "value-added" analysis is ever done, so people can have a 30 year teaching career without ever realizing that they aren't effective in the classroom. I've done many dozens of business presentations, to venture capitalists, technology partners, customers, analysts and even potential M&A acquirers, which has helped me improve my own teaching and communication skills. Despite the business setting such meetings are 90 percent teaching -- trying to convey key points to the audience in a limited time. I'm usually there with a team and my team isn't shy about telling me afterwards what worked and what didn't work, so I've had a lot of honest feedback that most professors never get.

The New Yorker cartoon and article capture some essential aspects of teaching and communication that are not widely understood. The teacher has to be simultaneously on top of the material itself and aware of what the class is doing / thinking / confused about. The big neglected factors in teaching are the ability to be a kind of air traffic controller (or symphony conductor) for the class, and the ability to empathize with (read the mind of) an individual student, to see what, exactly, is confusing them.


Anonymous said...

Is the difference between first and fourth quartile large enough that it can't be explained by chance?

It is a good thing that standardized tests are used to measure students. The US is almost alone in the whole world in relying on grades to sort people.
The result is that the executives, investment bankers, lawyers, professors, politicians, etc. of foreign countries have higher iqs than their us counterparts, and that pushiness is valued over all other characteristics.

Of course, the most effectve teacher is a machine.

hcl said...

K-12 may be where many/most/all students are better off with professionally produced videos featuring experts in the field or best-in-country teachers.

Why does the country require 100,000 math teachers to explain poorly (and expensively) each year what several gifted teachers can explain well once and forever?

Whereas the university professor actually is cutting edge in some ways (knowledgeable of latest research, specialization in at least one area, etc), the grade school teacher truly offers nothing.

Deane said...

Doesn't the Gladwell article beg the question of how *other* countries seem to be able to teach their children so much better than us?

I don't understand why Gladwell mentions how much better other countries do than the US, but then displays absolutely no curiosity about why.

Does he believe, for example, that other countries somehow identify the "naturally talented" teachers? Or are they, unlike us, able to teach ordinary human beings how to teach effectively?

Anonymous said...

hcl spot on. with the video player and the personal computer (what I meant by machines) there is no excuse for the enormous waste of money on "teachers" and classrooms.

but, of course, there has been no excuse for "teachers" since the printing press and the mass literacy it made possible.

i can honestly say that every moment i've spent in a classroom has been a waste of time. i have always beeen better off reading the text and doing the problems.

STS said...


Where *classroom* teaching is concerned, I think I can agree. It has never been an improvement over doing the reading on my own. But I've other kinds of teaching -- like piano lessons, or meetings with a thesis advisor -- which were very valuable.

Classrooms seem to enforce passivity. Even when interruptions are genuinely welcomed, you can't change the *pace* of a lecture with a question, or take it into an interesting digression, if the rest of the class are sitting there impatiently waiting for the train to start up again. The unspoken demand from the group is: get back in line and go back to sleep ;)

Anonymous said...

The analogy with software developers is striking. It has often been observed that the best developers are far better than average developers, and get much more done.

Anonymous said...

In Singapore, the students just crank out and/or memorize all the possible questions in those "Ten Year Series" books of old exams. It becomes an exercise in "pattern recognition".

Anonymous said...

and in singapore, i would have gone to your best (your only?)university in whatever subject i chose, but in the US where pushiness and obedience is valued over all else my 1560 on the SAT and >95th percentile on the CBATs and 99th on the ACT couldn't even get me in to the U of Oregon.

as the late great Bobby Fischer said, "the US is sh@t."

pt said...

1) I love bell curves!

2) I highly recomment Ken Bain's book "What the best college teachers do".

Bee said...

Interesting. Regarding your last sentence. I don't have a lot of teaching experience, but one problem I have repeatedly come across is that students were often not able to tell what their problem was. I would ask somebody what it is he doesn't understand, and he's say he doesn't understand anything. So I've tried to tell them from the very beginning it's okay not to understand but they have find out what it is they don't understand and how to ask questions. That worked for many, but there are still those who aren't able to formulate their problem. And I can kind of relate to this, because I myself am one of these people who don't understand anything, until really all information is in place. But how does one help people like that?

amckinstry said...

Interesting: can someone from a US background explain how they tested the effectiveness of teaching ?

AFAICT the Brookings report does not describe the students and testing methods; I can't tell if they are K-12 students (what we in the UK and Ireland call Primary school) or older (Secondary level).

Here (in Ireland) there is no formal testing in primary school (exams, etc). There are reports by teachers, but nothing like the exams you get in secondary level, on which a study like the Brookings one could be based.

The distinction is important: the teachers unions will emphasize the importance of class sizes especially for primary schools.

Secondly, the stats seem to depend a lot on the average change in the class, or mean student. Again, class sizes are said to adversely influence the _lower_ students who are struggling; good students will be blocked at the hands of a bad teacher who eg. doesn't cover the material or is bad at time management. The teachers unions will point out that class sizes are what afffects poor students: lack of one-to-one interaction when they get stuck and fall behind.

It would be interesting to see more detail in the report: how the students were tested, and the standard deviation as well as the mean results.

Anonymous said...

It could be worse, such as those "cram schools" in places like Japan, Korea, etc ...


Peter Love said...

``I've done many dozens of business presentations, to venture capitalists, technology partners, customers, analysts and even potential M&A acquirers, which has helped me improve my own teaching and communication skills. Despite the business setting such meetings are 90 percent teaching -- trying to convey key points to the audience in a limited time. I'm usually there with a team and my team isn't shy about telling me afterwards what worked and what didn't work, so I've had a lot of honest feedback that most professors never get.''

I echo this sentiment - in the last 2.5 years of teaching my limited experience of business presentations was so much more helpful than my limited teaching experience! (And the students are so much smarter than the MBA's :-)



brit com said...

To Bee:
I teach English to students from 7th to 12th grades in Portugal. One thing I have found useful is to divide the resolution of exercises into several smaller steps. This has been helpful in understanding which of the steps causes difficulties to each one of the students.

Blog Archive