Examples of Perverse Incentives and Replication in Science

In an earlier post (Perverse Incentives and Replication in Science), I wrote:

Here's a depressing but all too common pattern in scientific research:

1. Study reports results which reinforce the dominant, politically correct, narrative.
2. Study is widely cited in other academic work, lionized in the popular press, and used to advance real world agendas.
3. Study fails to replicate, but no one (except a few careful and independent thinkers) notices.

This seems to have hit a nerve, as many people have come forward with their own examples of this pattern.

From a colleague at MSU:

Parts 1 and 2: Green revolution in Malawi from Farm Input Subsidy Program? Hurrah! Gushing coverage in the NYTimes, Jeffrey Sachs claiming credit, etc.

http://www.nytimes.com/2012/04/20/opinion/how-malawi-fed-its-own-people.html
http://www.nytimes.com/2007/12/02/world/africa/02malawi.html
http://www.nytimes.com/slideshow/2007/12/01/world/20071202MALAWI_index.html

Part 3: Failed replication? No actual green revolution? Will anyone notice?

Re-evaluating the Malawian Farm Input Subsidy Programme (Nature Plants)

Joseph P. Messina1*†, Brad G. Peter1† and Sieglinde S. Snapp2†

Abstract: The Malawian Farm Input Subsidy Programme (FISP) has received praise as a proactive policy that has transformed the nation’s food security, yet irreconcilable differences exist between maize production estimates distributed by the Food and Agriculture Organization of the United Nations (FAO), the Malawi Ministry of Agriculture and Food Security (MoAFS) and the National Statistical Office (NSO) of Malawi. These differences illuminate yield-reporting deficiencies and the value that alternative, politically unbiased yield estimates could play in understanding policy impacts. We use net photosynthesis (PsnNet) as an objective source of evidence to evaluate production history and production potential under a fertilizer input scenario. Even with the most generous harvest index (HI) and area manipulation to match a reported error, we are unable to replicate post-FISP production gains. In addition, we show that the spatial delivery of FISP may have contributed to popular perception of widespread maize improvement. These triangulated lines of evidence suggest that FISP may not have been the success it was thought to be. Lastly, we assert that fertilizer subsidies may not be sufficient or sustainable strategies for production gains in Malawi.

Introduction: Input subsidies targeting agricultural production are frequent and contentious development strategies. The national scale FISP implemented in Malawi has been heralded as an ‘African green revolution’ success story1. The programme was developed by the Malawian government in response to long-term recurring food shortages, following the notably poor maize harvest of 2005; the history of FISP is well described by Chirwa and Dorward2. Scholars and press sources alike commonly refer to government statistics regarding production and yields as having improved significantly. Reaching widespread audiences, Sachs broadcasted that “production doubled within one harvest season” following its deployment3. The influential policy paper by Denning et al. opened with the statement that the “Government of Malawi implemented one of the most ambitious and successful assaults on hunger in the history of the African continent”4. The Malawi success narrative has certainly influenced global development agencies, resulting in increased support for agricultural input subsidies; Tanzania, Zambia, Kenya and Rwanda have all followed suit and implemented some form of input subsidy programme. There has been mild economic criticism of the subsidy implementation process, including disruption of private fertilizer distribution networks within the policy’s first year5. Moreover, the sustainability of subsidies in Malawi has been debated6,7, yet crop productivity gains from subsidies have gone largely unquestioned. As Sanchez commented, “in spite of criticisms by donor agencies and academics, the seed and fertilizer subsidies provided food security to millions of Malawians”1. This optimistic assessment of potential for an “African green revolution” must be tempered by the fact that the Malawian production miracle appears, in part, to be a myth. ...

For more on the 1-2-3 pattern and replication, see this blog post by economist Douglas Campbell, and discussion here.

For more depressing narrative concerning the reliability of published results (this time in cancer research), see this front page NYTimes story from today.