The term “p-hacking” has made its way into the public discourse surrounding science, particularly regarding the replicability crisis. But although the term suggests intentional malevolence on the part of the scientist, it’s actually a scenario that many well-trained scientists can fall into. So what is p-hacking, and why is it dangerous?
Science faces a replicability crisis. This is well-known among scientists and even the general public. Various explanations have been proposed as to the cause of this crisis – some of them on this blog – but these proposals have usually been informal in nature. Recently, two scientists (Smaldino & McElreath, 2016) built a computational model to explain the replicability crisis.
Science is a framework for understanding the world. We observe a phenomenon, ask questions about it, and build models to describe and predict it. Crucially, this process is iterative. We’re constantly refining our experiments, theories, and models, in an effort to improve our understanding of some phenomenon. Instead of accepting the results of a study as “fact”, we ask whether those results can be replicated reliably under similar and different conditions, what theories can be extrapolated from those results, and how we might extend those results to new domains or new questions.
Iteration is how science fosters dialogue. Without it, science isn’t really science; it’s just a bunch of people shouting over one another, with nobody listening. Or even worse, it’s like a game of telephone, in which nobody checks whether the last person got the message right.