Statistical reproducibility for pairwise t-tests in pharmaceutical research

Simkus, A.; Coolen, F.P.A.; Coolen-Maturi, T.; Karp, N.A.; Bendtsen, C.

doi:10.1177/09622802211041765

Statistical reproducibility for pairwise t-tests in pharmaceutical research

Simkus, A.; Coolen, F.P.A.; Coolen-Maturi, T.; Karp, N.A.; Bendtsen, C.

Authors

A. Simkus

Professor Frank Coolen frank.coolen@durham.ac.uk
Professor

Dr Tahani Coolen-Maturi tahani.maturi@durham.ac.uk
Associate Professor

N.A. Karp

C. Bendtsen

Abstract

This paper investigates statistical reproducibility of the t-test. We formulate reproducibility as a predictive inference problem and apply the nonparametric predictive inference (NPI) method. Within our research framework, statistical reproducibility provides inference on the probability that the same test outcome would be reached, if the test were repeated under identical conditions. We present an NPI algorithm to calculate the reproducibility of the t-test and then use simulations to explore the reproducibility both under the null and alternative hypotheses. We then apply NPI reproducibility to a real life scenario of a preclinical experiment, which involves multiple pairwise comparisons of test groups, where different groups are given a different concentration of a drug. The aim of the experiment is to decide the concentration of the drug which is most effective. In both simulations and the application scenario, we study the relationship between reproducibility and two test statistics, the Cohen’s d and the p-value. We also compare the reproducibility of the t-test with the reproducibility of the Wilcoxon Mann-Whitney test. Finally, we examine reproducibility for the final decision of choosing a particular dose in the multiple pairwise comparisons scenario. This paper presents advances on the topic of test reproducibility with relevance for tests used in pharmaceutical research.

Citation

Simkus, A., Coolen, F., Coolen-Maturi, T., Karp, N., & Bendtsen, C. (2022). Statistical reproducibility for pairwise t-tests in pharmaceutical research. Statistical Methods in Medical Research, 31(4), 673-688. https://doi.org/10.1177/09622802211041765

Journal Article Type	Article
Acceptance Date	Jul 31, 2021
Online Publication Date	Dec 2, 2021
Publication Date	Apr 1, 2022
Deposit Date	Jul 31, 2021
Publicly Available Date	Aug 2, 2021
Journal	Statistical Methods in Medical Research
Print ISSN	0962-2802
Electronic ISSN	1477-0334
Publisher	SAGE Publications
Peer Reviewed	Peer Reviewed
Volume	31
Issue	4
Pages	673-688
DOI	https://doi.org/10.1177/09622802211041765

Files

Accepted Journal Article (768 Kb)
PDF

Copyright Statement
This article is distributed under the terms of the Creative Commons Attribution 4.0 License (https://creativecommons.org/licenses/by/4.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access page (https://us.sagepub.com/en-us/nam/open-access-at-sage).