What is this thing called 'efficacy'?

Abstract

This paper is about efficacy, effectiveness, the need for theory to join the two, and the tragedies of exporting the Cochrane medical-inspired ideology to social policy. Loosely, efficacy is what is established about causes in RCTs – randomized controlled trials. Effectiveness is what a cause does ‘in the field’. The theory, like that describing forces in mechanics, underwrites the assumption that the cause contributes the same effect in the field as in the experiment. The tragedies are multiple and snowball on one another. On conventional Cochrane Collaboration doctrine, following the model of testing pharmaceuticals, the RCT is the gold standard for evidence of effectiveness in evidence-based policy. The first tragedy is that on dominant characterizations of ‘efficacy’, including, especially, many that try hard to be scientific, it does not make sense to suppose that efficacies make any difference outside experiments. The second tragedy is that once ‘efficacy’ is characterized so that it does make sense, the RCT can hardly be a gold standard since it goes no way towards establishing the theory, or more loosely the story or account, that it takes to get out of the experiment and into the field. The third tragedy is that much of the teaching about evidence-based policy pays little attention to the need for such theories or accounts. Indeed there is often the suggestion that RCTs should replace such accounts since the accounts are almost always controversial. The worst tragedy is that we offer advice that lets policy down, wasting the powerful knowledge that could be provided by RCTs. We pay heavily to measure efficacies in RCTs but efficacy is not evidence at all for effectiveness without the right kind of account or theory to make it so. Yet we provide hardly any guidance about how to manage when accounts are dicey.