One funeral at a time?

A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it. -- Max Planck

I'm at the annual AAU meeting of research Vice-Presidents, and the "reproducibility crisis" (in some fields) is one of the topics on the agenda. Today I heard a nice talk by Brian Nosek on the Reproducibility Project and Open Science Framework. Afterwards I asked him whether other social psychologists were really absorbing the implications of the low reproducibility rate in their field. Nosek said that attitudes were changing rapidly and that in 10 years the field would be very different. If so, progress can happen much faster than in Max Planck's pessimistic quote. Let's hope Nosek is right!

As an example of soul searching (learning can hurt!) Nosek pointed me to this blog post, by Professor Michael Inzlicht of Toronto. To his credit, Inzlicht is taking seriously the replication difficulties of two effects he has worked on in the past: ego depletion (i.e., willpower fatigue) and stereotype threat. Anyone who has looked carefully at the literature (positive effects from small sample sizes, but failed replication or very small effect size results from much larger samples) has to question whether these much-hyped phenomena are real. There is also lots of evidence for publication bias.

Inzlicht: ... I have spent nearly a decade working on the concept of ego depletion, including work that is critical of the model used to explain the phenomenon. I have been rewarded for this work, and I am convinced that the main reason I get any invitations to speak at colloquia and brown-bags these days is because of this work. The problem is that ego depletion might not even be a thing. By now, many people are aware that a massive replication attempt of the basic ego depletion effect involving over 2,000 participants found nothing, nada, zip. Only three of the 24 participating labs found a significant effect, but even then, one of these found a significant result in the wrong direction!

There is a lot more to this registered replication than the main headline, and deep in my heart, it is hard to believe that fatigue is not a real phenomenon. I promise to get to it in a later post. But for now, we are left with a sobering question: If a large sample pre-registered study found absolutely nothing, how has the ego depletion effect been replicated and extended hundreds and hundreds of times? More sobering still: What other phenomena, which we now consider obviously real and true, will be revealed to be just as fragile?

As I said, I’m in a dark place. I feel like the ground is moving from underneath me and I no longer know what is real and what is not.

I edited an entire book on stereotype threat, I have signed my name to an amicus brief to the Supreme Court of the United States citing stereotype threat, yet now I am not as certain as I once was about the robustness of the effect. I feel like a traitor for having just written that; like, I’ve disrespected my parents, a no no according to Commandment number 5. But, a meta-analysis published just last year suggests that stereotype threat, at least for some populations and under some conditions, might not be so robust after all. P-curving some of the original papers is also not comforting. Now, stereotype threat is a politically charged topic and I really really want it to be real. I think a lot more pain-staking work needs to be done before I stop believing (and rumor has it that another RRR of stereotype threat is in the works), but I would be lying if I said that doubts have not crept in. ...

See also Is science self-correcting? Figure at top is from this meta-analysis of stereotype threat.