Replication and the "Crises of Confidence" in Science

One of the authors pointed me to the interesting paper below, which contains a proposal meant to improve the reliability of scientific research (specifically in some areas such as social science or biomedicine). Could this proposal work if, say, it were strongly supported by scholarly associations and funding agencies?

Note the authors' use of "Crises of Confidence" in their title. I believe that certain areas of science really should be experiencing a crisis of confidence, if only their practitioners were smarter and more honest. But in fact I suspect it's mostly business as usual, even in the research areas with the lowest (and best documented as lowest) replication rates.

An Economic Approach to Alleviate the Crises of Confidence in Science: With An Application to the Public Goods Game

Luigi Butera, John A. List  (University of Chicago)

April 5, 2017

... This paper proposes and puts into practice a novel and simple mechanism that allows mutually beneficial gains from trade between original investigators and other researchers. In our mechanism, the original investigators, upon completing their initial study, write a working paper version of their study. While they do share their working paper online, they do however commit not to submit it to any journal for publication, ever. The original investigators instead offer co-authorship of a second paper to other researchers who are willing to independently replicate the experimental protocol in their own research facilities.2 Once the team is established, but before beginning replications, the replication protocol is pre-registered at the AEA experimental registry, and referenced in the first working paper. This is to guarantee that all replications, both successful and failed, are properly accounted for, eliminating any concerns about publication biases. The team of researchers composed by the original investigators and the other scholars will then write and coauthor a second paper, which will reference the original unpublished working paper, and submit it to an academic journal. Under such an approach, the original investigators accept to publish their work with several other coauthors, a feature that is typically unattractive to economists, but in turn gain a dramatic increase in the credibility and robustness of their results, should they replicate. Further, the referenced working paper would provide a credible signal about the ownership of the initial research design and idea, a feature that is particularly desirable for junior scholars. On the other hand, other researchers would face the monetary cost of replicating the original study, but would in turn benefit from coauthoring a novel study, and share the related payoffs. Overall, our mechanism could critically strengthen the reliability of novel experimental results and facilitate the advancement of scientific knowledge.