Where is the rigor when you need it?

Abstract

When it comes to causal conclusions, rigor matters. To this end we impose high standards for how studies from which we draw causal conclusions are conducted. For instance, we are widely urged to prefer randomized controlled trials (RCTs) or instrumental variable (IV) models to observational studies relying just on correlations, and we have explicit criteria for what counts as a good RCT or a good IV model. But we tend to be shockingly sloppy when it comes to making explicit just what the causal conclusions we draw mean, why the methods we employ are good for establishing conclusions with just that meaning, and what can defensibly be taken to follow from these claims. With respect to what can be inferred from the limited causal conclusions our studies support, we are far too prone to over reach, to ‘generalize’ that what holds in a study or handful of studies holds widely. But, I shall argue, we do not get arrant for general claims by generalizing. Rather it takes a great tangle of scientific work to support a general claim, including a great deal of conceptual development, theory and the confirmation of a variety of different kinds of effects that the general claim implies.