Actor-Driven Risk Factors of Publication Bias: Opening the File Drawer of Two Probabilistic Panel Surveys

Abstract

Science has long struggled with unsuccessful replications and reproductions, a challenge that raises questions about the robustness and credibility of research findings. One potential contributing factor is selective reporting or nonpublication of certain results based on their direction or strength, leading to publication bias. To better understand its extent and risk factors, we examined 178 successful study submissions to two German probabilistic panels (GESIS Panel, SOEP-IS), fielded between 2013 and 2021, and compared the hypotheses and exploratory analyses stated in the original study submissions with their presentation in subsequent publications. Using a coding scheme, we extracted information on the submissions, publications, and authors, and investigated which characteristics are associated with publication bias. Experimental study design, third-party funding, preregistration of hypotheses, and economic studies were the strongest and most stable predictors of lower risk of publication bias across multiple outcomes: publication likelihood, test and hit rate of hypotheses. The study highlights the complexity and multi-faceted nature of publication bias in the social, economic, and behavioral sciences, and shows that researchers (e.g., preregistration), funding agencies (e.g., incentives), and journals (e.g., peer review) play distinct roles in shaping publication bias. It thereby points to areas for reform in the research and publication process.