In general, these procedures are acceptable when transparently reported however, when authors neglect to accurately report how the results were in fact generated, they are rightfully classified as data-dredging. In such cases an unconscious tendency to interpret the results in a biased fashion can be guarded against by prespecifying the course of analysis.Īpart from p-hacking, other forms of data dredging include: assessing models with multiple combinations of variables and selectively reporting the “best” model (i.e., “fishing”) making decisions about whether to collect new data on the basis of interim results making post-hoc decisions about which statistical analyses to conduct and generating a hypothesis to explain results which have already been obtained but presenting it as it were a hypothesis one had prior to collecting the data (i.e., HARKing (“hypothesizing after the results are known”)).

As Gelman and Loken (2014) note, “it can seem entirely appropriate to look at the data and construct reasonable rules for data exclusion, coding, and analysis that can lead to statistical significance” (p.

Though some forms of data-dredging are lamentably common, it is important to note that often such problems arise from a lack of awareness rather than malfeasance. ‘false positives’) and is thus unreliable. Such an analysis can often generate statistically significant results in absence of a true effect (i.e. While many different choices might be defensible, a canonical case of p-hacking would involve trying out multiple different options and reporting the result which yields the lowest p-value (particularly when alternative choices generate values that do not yield a significant result). how to handle outliers, whether to combine groups, including/excluding covariates) which will produce a statistically significant p-value. In contrast, p-hacking occurs when an initial analysis produces results which are close to being statistically significant, then, in absence of a study protocol, researchers can make analytic choices (e.g. Ideally, these choices are guided by the principles of best practice and prespecified in a publicly available protocol. For example, in nearly any analysis of data there are several “researcher degrees of freedom”- i.e., choices that must be made in the process of analysis. fishing, p-hacking), but each essentially involves probing the data in unplanned ways, finding and reporting an “attractive” result, without accurately conveying the course of analysis. Data-dredging bias is a general category which includes a number of misuses of statistical inference (e.g.

Data dredging/ p-hacking