An Empirical Unravelling of Lord’s Paradox

Xiao, Z.; Higgins, S.; Kasim, A.

doi:10.1080/00220973.2017.1380591

An Empirical Unravelling of Lord’s Paradox

Xiao, Z.; Higgins, S.; Kasim, A.

Authors

Z. Xiao

Professor Steven Higgins s.e.higgins@durham.ac.uk
Professor

A. Kasim

Abstract

Lord's Paradox occurs when a continuous covariate is statistically controlled for and the relationship between a continuous outcome and group status indicator changes in both magnitude and direction. This phenomenon poses a challenge to the notion of evidence-based policy, where data are supposed to be self-evident. We examined 50 effect size estimates from 34 large-scale educational interventions and found that impact estimates are affected in magnitude, with or without reversal in sign, when there is substantial baseline imbalance. We also demonstrated that multilevel modeling can ameliorate the divergence in sign and/or magnitude of effect estimation, which, together with project specific knowledge, promises to help those who are presented with conflicting or confusing evidence in decision-making.

Citation

Xiao, Z., Higgins, S., & Kasim, A. (2019). An Empirical Unravelling of Lord’s Paradox. The Journal of Experimental Education, 87(1), 17-32. https://doi.org/10.1080/00220973.2017.1380591

Journal Article Type	Article
Acceptance Date	Sep 6, 2017
Online Publication Date	Nov 7, 2017
Publication Date	2019
Deposit Date	Jul 17, 2017
Publicly Available Date	May 7, 2019
Journal	Journal of Experimental Education
Print ISSN	0022-0973
Electronic ISSN	1940-0683
Publisher	Taylor and Francis Group
Peer Reviewed	Peer Reviewed
Volume	87
Issue	1
Pages	17-32
DOI	https://doi.org/10.1080/00220973.2017.1380591
Keywords	Lord's Paradox, RCT, Multilevel Modelling, Evidence-Based Policy, Evaluation.
Related Public URLs	https://osf.io/auq24/

Files

Accepted Journal Article (Revised version) (398 Kb)
PDF

Copyright Statement
Revised version This is an Accepted Manuscript of an article published by Taylor & Francis in Journal of Experimental Education on 07 Nov 2017, available online: http://www.tandfonline.com/10.1080/00220973.2017.1380591.