Abstract

P-hacking involves the manipulation of data to find a statistically significant result. Many claim that p-hacking is a problem in science, especially in the medical and social sciences, while others deny this. The problem with p-hacking is usually articulated from a frequentist perspective. In this paper we articulate the epistemic peril of p-hacking using Bayesian confirmation theory and model selection theory, which we then draw on to explain the arguments on both sides of the debate. This requires a novel understanding of Bayesianism, since a standard criticism of Bayesian confirmation theory is that it cannot accommodate the influence of biased methods. A methodological device widely used to mitigate the peril of p-hacking is a pre-analysis plan. Some say that following a pre-analysis plan is epistemically meritorious while others deny this, and in practice pre-analysis plans are often violated. We use the formal groundwork developed earlier in the paper to resolve this debate, offering a modest defence of the use of pre-analysis plans. In the longer run our ambition is to use this approach to make sense of scenarios in which scientists depart from pre-analysis plans.