A well-known problem in classical two-tailed hypothesis testing is that P-values go to zero when the sample size goes to infinity, irrespectively of the effect size. This pitfall can make the testing of data consisting of large sample sizes potentially unreliable. In this note, we propose to test for relevant differences to overcome this issue. We illustrate the proposed test a on real data set of about 40 million privately insured patients.
Callegaro, A., Ndour, C., Aris, E., & Legrand, C. (2019). A note on tests for relevant differences with extremely large sample sizes. Biometrical Journal : journal of mathematical methods in biosciences, 61(1), 162-165. https://doi.org/10.1002/bimj.201800195 (Original work published 2019)