Discussion
Tuesday, May 9 |
4:30 PM
The Promise and Pitfalls of Multiverse Analysis: A Moderated Discussion
Abstract

Science is going meta. In a ‘reproducibility crisis,’ scholars are responding with many-analyst studies, meta-analysis, and multiverse research. Science is based on hard data, but also struggles with the inherent subjectivity of researchers. There is growing demand for robustness, with the concern that any researcher could potentially arrive at any result in a single analysis – lost in the garden of forking paths and researcher degrees of freedom. Multiverse analysis aspires to an ideal of many models in a world of theory competition. This has led to simulations of plausible models for a multiverse of potential results. Whether results were generated by many independent researchers or via simulation of plausible data-generating models, the question arises of how to produce, analyze, parse and understand the hundreds, thousands or even millions of results. While the spirit is strong, the exact methods of multiverse analysis are often rudimentary and searching for greater expression and refinement. The project is making halting progress towards greater robustness in science. In this moderated discussion we bring together experts to discuss the future of multiverse methods and the challenges and pitfalls it faces.