Multiple recent ‘many-analyst’ projects have demonstrated that when different analysts seek to answer the same question with the same data, they often produce a substantially heterogeneous set of answers. We evaluated the extent of this phenomenon in ecology and evolutionary biology with a registered report in which several hundred biologists analyzed data from either of two data sets to answer a predetermined question associated with each data set. We found substantial heterogeneity in results from these analyses, but this heterogeneity differed qualitatively and quantitatively between the two data sets. Surprisingly, very little of the observed heterogeneity in results among analyses appeared related to the assessments of the individual analyses submitted by peer reviewers. Our results have major implications for how ecologists and evolutionary biologists conduct and interpret statistical analyses.