The Problem of New Evidence: P-Hacking and Pre-Analysis Plans

Main Article Content

Zoe Hitzig
https://orcid.org/0000-0002-1589-2318
Jacob Stegenga
https://orcid.org/0000-0002-7016-3708

Abstract

We provide a novel articulation of the epistemic peril of p-hacking using three resources from philosophy: predictivism, Bayesian confirmation theory, and model selection theory. We defend a nuanced position on p-hacking: p-hacking is sometimes, but not always, epistemically pernicious. Our argument requires a novel understanding of Bayesianism, since a standard criticism of Bayesian confirmation theory is that it cannot represent the influence of biased methods. We then turn to pre-analysis plans, a methodological device used to mitigate p-hacking. Some say that pre-analysis plans are epistemically meritorious while others deny this, and in practice pre-analysis plans are often violated. We resolve this debate with a modest defence of pre-analysis plans. Further, we argue that pre-analysis plans can be epistemically relevant even if the plan is not strictly followed—and suggest that allowing for flexible pre-analysis plans may be the best available policy option.

Downloads

Download data is not yet available.

Article Details

How to Cite
Hitzig, Zoe, and Jacob Stegenga. 2020. “The Problem of New Evidence: P-Hacking and Pre-Analysis Plans”. Diametros 17 (66):10-33. https://doi.org/10.33392/diam.1587.
Section
Articles
Share |

References

Backhouse R.E., Morgan M.S. (2000), “Introduction: is Data Mining a Methodological Problem?,” Journal of Economic Methodology 7 (2): 171–181.
View in Google Scholar
DOI: https://doi.org/10.1080/13501780050045065

Barnes E.C. (2008), The Paradox of Predictivism, Cambridge University Press, Cambridge.
View in Google Scholar
DOI: https://doi.org/10.1017/CBO9780511487330

Bright L.K. (2017), “On Fraud,” Philosophical Studies 174 (2): 291–310.
View in Google Scholar
DOI: https://doi.org/10.1007/s11098-016-0682-7

Brodeur A., Lé M., Sangnier M. et al. (2016), “Star Wars: The Empirics Strike Back,” American Economic Journal: Applied Economics 8 (1): 1–32.
View in Google Scholar
DOI: https://doi.org/10.1257/app.20150044

Camerer C.F., Dreber A., Holzmeister F. et al. (2018), “Evaluating the Replicability of Social Science Experiments in Nature and Science Between 2010 and 2015,” Nature Human Behaviour 2 (9): 637–644.
View in Google Scholar
DOI: https://doi.org/10.1038/s41562-018-0399-z

Casey K., Glennerster R., Miguel E. (2012), “Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan,” The Quarterly Journal of Economics 127 (4): 1755–1812.
View in Google Scholar
DOI: https://doi.org/10.1093/qje/qje027

Chambers C.D. (2013), “Registered Reports: A New Publishing Initiative at Cortex,” Cortex 49 (3): 609–610.
View in Google Scholar
DOI: https://doi.org/10.1016/j.cortex.2012.12.016

Chambers C.D., Feredoes E., Muthukumaraswamy S.D. et al. (2014), “Instead of ‘Playing the Game’ it is Time to Change the Rules: Registered Reports at AIMS Neuroscience and Beyond,” AIMS Neuroscience 1 (1): 4–17.
View in Google Scholar
DOI: https://doi.org/10.3934/Neuroscience.2014.1.4

Christensen G., Miguel E. (2018), “Transparency, Reproducibility, and the Credibility of Economics Research,” Journal of Economic Literature 56 (3): 920–980.
View in Google Scholar
DOI: https://doi.org/10.1257/jel.20171350

Coffman L.C., Niederle M. (2015), “Pre-Analysis Plans have Limited Upside, Especially where Replications are Feasible,” Journal of Economic Perspectives 29 (3): 81–98.
View in Google Scholar
DOI: https://doi.org/10.1257/jep.29.3.81

Diaconis P., Mosteller F. (1989), “Methods for Studying Coincidences,” Journal of the American Statistical Association 84 (408): 853–861.
View in Google Scholar
DOI: https://doi.org/10.1080/01621459.1989.10478847

Douglas H., Magnus P.D. (2013), “State of the Field: Why Novel Prediction Matters,” Studies in History and Philosophy of Science Part A 44 (4): 580–589.
View in Google Scholar
DOI: https://doi.org/10.1016/j.shpsa.2013.04.001

Dwan K., Altman D., Arnaiz J. et al. (2008), “Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias,” PLoS ONE 3 (8): e3081.
View in Google Scholar
DOI: https://doi.org/10.1371/journal.pone.0003081

FDA (1997), “Food and Drug Administration Modernization Act,” 105th U.S. Congress, U.S. House of Representative Bill, URL = https://www.congress.gov/bill/105th-congress/senate-bill/830 [Accessed 13.07.2020].
View in Google Scholar

Findley M.G., Jensen N.M., Malesky E.J. et al. (2016), “Can Results-Free Review Reduce Publication Bias? The Results and Implications of a Pilot Study,” Comparative Political Studies 49 (13): 1667–1703.
View in Google Scholar
DOI: https://doi.org/10.1177/0010414016655539

Foster A., Karlan D., Miguel E. (2018), “Registered Reports: Piloting a Pre-Results Review Process at the Journal of Development Economics,” World Bank Development Impact Blog, URL = https://blogs.worldbank.org/impactevaluations/registered-reportspiloting-pre-results-review-process-journal-development-economics [Accessed 02.07.2020].
View in Google Scholar

Frankel A., Kasy M. (2020), “Which Findings should be Published?,” URL = https://maxkasy.github.io/home/files/papers/findings.pdf [Accessed 25.08.2020].
View in Google Scholar

Frisch M. (2015), “Predictivism and Old Evidence: A Critical Look at Climate Model Tuning,” European Journal for Philosophy of Science 5 (2): 171–190.
View in Google Scholar
DOI: https://doi.org/10.1007/s13194-015-0110-4

Glymour C. (1980), Theory and Evidence, Princeton University Press, Princeton, N.J.
View in Google Scholar

Head M.L., Holman L., Lanfear R. et al. (2015), “The Extent and Consequences of P-Hacking in Science,” PLoS Biology 13 (3): e1002106.
View in Google Scholar
DOI: https://doi.org/10.1371/journal.pbio.1002106

Howson C. (1991), “The ‘Old Evidence’ Problem,” The British Journal for the Philosophy of Science 42 (4): 547–555.
View in Google Scholar
DOI: https://doi.org/10.1093/bjps/42.4.547

Howson C., Franklin A. (1991), “Maher, Mendeleev and Bayesianism,” Philosophy of Science 58 (4): 574–585.
View in Google Scholar
DOI: https://doi.org/10.1086/289641

Humphreys M., Sanchez De la Sierra R., Windt P.V.D. (2013), “Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration,” Political Analysis 21 (1): 1–20.
View in Google Scholar
DOI: https://doi.org/10.1093/pan/mps021

Ioannidis J.P.A. (2005), “Why Most Published Research Findings are False,” PLoS Medicine 2 (8): e124.
View in Google Scholar
DOI: https://doi.org/10.1371/journal.pmed.0020124

Ioannidis J.P.A. (2008), “Why Most Discovered True Associations are Inflated,” Epidemiology 19 (5): 640–648.
View in Google Scholar
DOI: https://doi.org/10.1097/EDE.0b013e31818131e7

Leamer E.E. (1983), “Let’s Take The Con out of Econometrics,” American Economic Review 73 (1): 31–43.
View in Google Scholar

Leonelli S. (2016), Data-Centric Biology: A Philosophical Study, University of Chicago Press, Chicago.
View in Google Scholar
DOI: https://doi.org/10.7208/chicago/9780226416502.001.0001

Libgober J. (Forthcoming), “False Positives and Transparency,” American Economic Journal: Microeconomics, URL = https://www.aeaweb.org/articles?id=10.1257/mic.20190218 [Accessed 01.10.2020].
View in Google Scholar

Maher P. (1988), “Prediction, Accommodation, and the Logic of Discovery,” [in:] PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association, vol. 1, A. Fine, J. Leplin (eds.), Philosophy of Science Association, East Lansing, MI: 273–285.
View in Google Scholar

Mayo D.G. (1996), Error and the Growth of Experimental Knowledge, University of Chicago Press, Chicago.
View in Google Scholar
DOI: https://doi.org/10.7208/chicago/9780226511993.001.0001

Miguel E., Camerer C., Casey K. et al. (2014), “Promoting Transparency in Social Science Research,” Science 343 (6166): 30–31.
View in Google Scholar
DOI: https://doi.org/10.1126/science.1245317

Nosek B.A., Lakens D. (2014), “Registered Reports: A Method to Increase the Credibility of Published Results,” Social Psychology 45 (3): 137–141.
View in Google Scholar
DOI: https://doi.org/10.1027/1864-9335/a000192

Olken B.A. (2015), “Promises and Perils of Pre-Analysis Plans,” Journal of Economic Perspectives 29 (3): 61–80.
View in Google Scholar
DOI: https://doi.org/10.1257/jep.29.3.61

Pagan A. (1987), “Three Econometric Methodologies: A Critical Appraisal,” Journal of Economic Surveys 1 (1–2): 3–23.
View in Google Scholar
DOI: https://doi.org/10.1111/j.1467-6419.1987.tb00022.x

Phillips P.C.B. (1988), “Reflections on Econometric Methodology,” Economic Record 64 (4): 344–359.
View in Google Scholar
DOI: https://doi.org/10.1111/j.1475-4932.1988.tb02075.x

Pearson T.A., Manolio T.A. (2008), “How to Interpret a Genome-Wide Association Study,” Journal of the American Medical Association 299 (11): 1335–1344.
View in Google Scholar
DOI: https://doi.org/10.1001/jama.299.11.1335

Simmons J.P., Nelson L.D., Simonsohn U. (2011), “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” Psychological Science 22 (11): 1359–1366.
View in Google Scholar
DOI: https://doi.org/10.1177/0956797611417632

Sober E. (2015), Ockham’s Razors. A User’s Manual, Cambridge University Press, Cambridge.
View in Google Scholar
DOI: https://doi.org/10.1017/CBO9781107705937

White R. (2003), “The Epistemic Advantage of Prediction over Accommodation,” Mind 112 (448): 653–683.
View in Google Scholar
DOI: https://doi.org/10.1093/mind/112.448.653

Worrall J. (2014), “Prediction and Accommodation Revisited,” Studies in History and Philosophy of Science Part A 45 (1): 54–61.
View in Google Scholar
DOI: https://doi.org/10.1016/j.shpsa.2013.10.001