A recipe for disappointment: policy, effect size and the winner’s curse

Abstract

Evidence-based education aims to support policy makers choosing between potential interventions. This rarely involves considering each in isolation; instead, sets of evidence regarding many potential policy interventions are considered. Filtering a set on any quantity measured with error risks the ‘winner’s curse’: conditional on selecting higher valued measures, the measurement likely overestimates the latent value. This paper explains the winner’s curse, illustrates it for one constrained and complete set of educational trials – the UK’s Education Endowment Foundation’s projects, where evidence is summarized with standardized effect size – and shows the results of adjusting for the curse on this set. This analysis suggests selecting policies for higher effect size can result in substantial effect size inflation and in some cases order reversals: one intervention ranking above another on estimated effect size but below it when adjusted. The issue has implications for evaluation programs, power analyses and policy decisions. For example, even in the absence of other problems with interpreting effect size, it can help explain why policies tend to deliver less than promised.