Join here: https://aarhusuniversity.zoom.us/j/65548867322

New 'many analyst' forms of research find that different researchers come to different results when analyzing the same hypothesis and data. The obvious reason is different steps taken in the research process. Presumably, tracing all steps will allow us to reliably assess outcome variability produced by this emerging form of meta-science. We tested this in a controlled setting with 73 research teams, observing each team’s workflow in producing a combined 1,261 analytical models and 89 hypothesis-target conclusions. We expected 'major' research design steps like measurement and subsetting, would explain a substantial portion of outcome variance. However, multivariate, multilevel and meta-analytic testing revealed that all steps explained at most 2.6% of the total effect size variance, and not more than 10% of the deviance in subjective conclusions. The expertise, prior beliefs and attitudes of researchers explained even less. Ultimately, 107 possible steps across teams meant that each model was entirely unique. Only through careful observation do we witness this vast universe of research design variability normally hidden from view in the presentation, consumption, and perhaps even creation of scientific results.

Read about the Aarhus Reproducibility and Open Science Seminars here.

Sign up below to recieve a reminder prior to this event.