Some basic observations on conducting a systematic review: A brief reply to Goldstein, Vatalaro and Yair

Abstract

Purpose: The purpose of this paper is to provide a response to Goldstein et al.’s (2017) attempted rebuttal of the authors’ prior paper in this journal (See and Gorard 2015). Design/methodology/approach: The prior paper reported a systematic review of interventions to involve engaging parents more in their children’s education in order to raise school attainment. Goldstein et al. make a large number of unwarranted claims about its quality. They reproach the authors for using reports of unpublished evidence, for mis-labelling or mis-describing studies, and for denigrating studies by labelling them as “bad”. The authors were very surprised when first alerted to this response and went back to look at all of the research reports that Goldstein et al. claimed the authors mis-represented in the authors’ assessment. Findings: The authors found that the Goldstein et al. claims are false and based on such a poor understanding of how evidence is reviewed that it was strange to see their paper in this journal. Originality/value: In the authors’ reply, they look first at why unpublished material must be included in a review, and why the outlet for publication is not relevant, then at appropriate designs for causal questions, and at the confusion in Goldstein et al. between evaluation quality and intervention impact. The authors look at many examples where the confusion leads to Goldstein et al. making incorrect assertions about the authors’ paper, in order to make the point that their whole idea of how to conduct a systematic review is wrong.