Nonparametric predictive inference for test reproducibility by sampling future data orderings

Abstract

This paper considers nonparametric predictive inference (NPI) for reproducibility of likelihood ratio tests with the test criterion in terms of the sample mean. Given a sample of size n used for the actual test, the NPI approach provides lower and upper probabilities for the event that a repeat of the test, also with n observations, will lead to the same overall test conclusion, that is rejecting a null-hypothesis or not. This is achieved by considering all orderings of n future observations among the n data observations, which based on an exchangeability assumption are equally likely. However, exact lower and upper probabilities can only be derived for relatively small values of n due to computational limitations. Therefore, the main aim of this paper is to explore sampling of the orderings of the future data among the observed data in order to approximate the lower and upper reproducibility probabilities. The approach is applied for the Exponential and Normal distributions, and the performance of the ordering sampling for approximation of the NPI lower and upper reproducibility probabilities is investigated. An application with real data of the methodology developed is provided.