An Investigation of <i>p</i>-Hacking in E-Commerce A/B Testing
Authors: Miller, Alex P.; Hosanagar, Kartik
Journal: Information Systems Research (2025)
<jats:p> Concerns about the integrity of statistical analyses have risen in recent years, with particular attention given to “p-hacking”—a process whereby analysts conduct several statistical tests until they achieve a statistically significant result. Although extensively studied in academic settings, less is known about its prevalence in industrial contexts. In this study, we investigate whether p-hacking occurs in e-commerce A/B testing. We analyzed nearly 2,300 A/B tests conducted by hundreds of firms using a large A/B testing platform. Such platforms typically offer continuous monitoring of test results, a feature that facilitates real-time decision making but also enables potential p-hacking through selective stopping or continuation of experiments. Contrary to concerns raised by earlier research on academic practices, we found no significant evidence of p-hacking in our sample. These findings suggest that the industrial application of experimentation may be less susceptible to p-hacking than academic research. We discuss several possible factors explaining the divergent results, highlighting the potential role of organizational learning and the importance of economic incent…