Investigating the analytical robustness of the social and behavioural sciences
(2026) Nature — Vol. 652, n° 8108, p. 135-142 (2026)
Files
s41586-025-09844-9-1.pdf
Embargo Access until 2027-04-01 - Adobe PDF
- 8.01 MB
Details
- Authors
Aczel, Balazs 
Szaszi, Barnabas Clelland, Harry
- Abstract
- Article Investigating the analytical robustness of the social and behavioural sciences The same dataset can be analysed in different justifiable ways to answer the same research question, potentially challenging the robustness of empirical science 1-3. In this crowd initiative, we investigated the degree to which research findings in the social and behavioural sciences are contingent on analysts' choices. We examined a stratified random sample of 100 studies published between 2009 and 2018, in which, for one claim per study, at least five reanalysts independently reanalysed the original data. The statistical appropriateness of the reanalyses was assessed in peer evaluations, and the robustness indicators were inspected along a range of research characteristics and study designs. We found that 34% of the independent reanalyses yielded the same result (within a tolerance region of ±0.05 Cohen's d) as the original report; with a four times broader tolerance region, this indicator increased to 57%. Of the reanalyses conducted, 74% reached the same conclusion as the original investigation, 24% yielded no effects or inconclusive results and 2% reported the opposite effect. This exploratory study indicates that the common single-path analyses in social and behavioural research should not be simply assumed to be robust to alternative analyses 4. Therefore, we recommend the development and use of practices to explore and communicate this neglected source of uncertainty. Over the past decade, social and behavioural scientists have been striving to enhance the robustness, objectivity and replicability of their findings through systemic reforms in the conduct and communication of empirical research. Practices such as preregistration 5 , registered reports 6 , multisite replications 7 , analytical reproducibility checks 8,9 and automated result validation techniques 10 have been investigated and recommended to produce robust and replicable findings. An important aspect of robustness has yet to be systematically charted across these sciences: the contingency of the results on researchers' analytical choices. In a typical research pipeline, the collected empirical data are analysed by a single analyst or team, and the published report presents a conclusion on the basis of one analytical path, occasionally accompanied by a few robustness tests. The peer review process aims to ensure that the analysis approach meets the relevant statistical and field-specific standards. However, this procedure does not systematically ascertain whether justifiable alternative analytical choices could have led to different results. Theories and empirical designs rarely constrain analysts to a single analytical path. Many degrees of freedom exist in how researchers operationalize their variables, process their data, construct their statistical models, select algorithms and software for model estimation and define their inference criteria, whether they follow frequentist, Bayesian or likelihoodist analytical approaches; use machine learning; or conduct computational modelling to answer the same research question 1,4. This inherent freedom of the analyst constitutes the so-called analytical variability contained within empirical projects, a key component in the robustness of the statistical results. In practical terms, it is the manifested variation among the choices independent scientists consider justified. Figure 1 presents some sources of analytical variability that can manifest themselves in analysts' statistical results and the conclusions drawn from the results. One way to explore analytical variability is to use a multiverse methodology 2,11 , in which analysts conduct all combinations of analytical choices they are able to generate across a wide range of reasonable scenarios. Alternatively, in the multi-analyst approach, several analysts analyse the data following their best judgement. The latter approach requires more organization, but it takes advantage of alternative expert perspectives without the combinatory expansion of the number of results. A multi-analyst approach also examines naturally occurring variation, empirically answering the counterfactual question of what might have happened if another investigator had considered the same research question using the same data. Multi-analyst projects 3,12-24 have provided some evidence of the extent to which analysts' individual choices influence results and conclusions. From economics to neuroscience, these explorations have demonstrated that the robustness of empirical findings can be compromised by researcher degrees of freedom 25. The estimates of previous multi-analyst studies suggest that the variability in effect-size estimates attributable to analytical heterogeneity can exceed the variability one would expect owing to sampling error 26. Do we know how robust published findings are to analytical choices across the social and behavioural sciences? One could argue that multi-analyst projects so far have been purposefully conducted in research areas with little consensus on the best analytical approach or were motivated to demonstrate the potential effects of analytical choices and, therefore, may represent rare cases in which alternative analyses produce important differences in results. For example, the datasets selected may have afforded researchers greater degrees of freedom than is typical, raising issues about the generalizability of
- Affiliations
Citations
Aczel, B., Szaszi, B., Clelland, H., Kovacs, M., Holzmeister, F., Van Ravenzwaaij, D., Schulz-Kümpel, H., Hoffmann, S., Nilsonne, G., Kosa, L., Torma, Z., Abdelfatah, Y., Aberson, C., Acar, O., Acem, E., Adamkovic, M., Adamovich, T., Adiasto, K., Ahnström, L., et al. (2026). Investigating the analytical robustness of the social and behavioural sciences. Nature, 652(8108), 135-142. https://doi.org/10.1038/s41586-025-09844-9 (Original work published 2026)