Predicting 'it will work for us': (way) beyond statistics.

Abstract

A great deal of attention in evidence‐based policy and practice is directed to statistical studies–especially randomized controlled trials–that support causal conclusions, which this chapter dubs ‘It‐works‐somewhere claims’. What's needed for policy and practice, however, are conclusions that the policy will work for us, as when and how we would implement it. Despite widespread recognition of the problem of external validity, it is all too easy to suppose that conclusions of the first sort provide strong evidence for those of the second sort. This chapter argues that this is not the case. Further, ‘external validity’ is the wrong way to characterize the problem. Usually the only reliable way to use an it‐works‐somewhere result as evidence for ‘It will work for us’ is via what J.S. Mill calls a ‘tendency’ claim (and the chapter calls a ‘capacity’ claim). This however points out how weak ‘It works somewhere’ is in support of ‘It will work for us’, for two reasons. (1) It takes a great deal of theory, observation and experiment, far beyond the statistical study itself, to establish a tendency/capacity claim; (2) Reliable prediction requires in addition a great deal of local knowledge supplied by neither the statistical study nor the capacity claim.