Skip to main content

Research Repository

Advanced Search

What to do instead of significance testing? Calculating the ‘number of counterfactual cases needed to disturb a finding’

Gorard, S.; Gorard, J.

What to do instead of significance testing? Calculating the ‘number of counterfactual cases needed to disturb a finding’ Thumbnail


Authors



Abstract

This brief paper introduces a new approach to assessing the trustworthiness of research comparisons when expressed numerically. The ‘number needed to disturb’ a research finding would be the number of counterfactual values that can be added to the smallest arm of any comparison before the difference or ‘effect’ size disappears, minus the number of cases missing key values. This way of presenting the security of findings has several advantages over the use of significance tests, effect sizes and confidence intervals. It is not predicated on random sampling, full response or any specific distribution of data. It bundles together the sample size, magnitude of the finding and the level of attrition in a way that is standardised and therefore comparable between studies.

Citation

Gorard, S., & Gorard, J. (2016). What to do instead of significance testing? Calculating the ‘number of counterfactual cases needed to disturb a finding’. International Journal of Social Research Methodology, 19(4), 481-490. https://doi.org/10.1080/13645579.2015.1091235

Journal Article Type Article
Acceptance Date Aug 23, 2015
Online Publication Date Oct 6, 2015
Publication Date Jul 1, 2016
Deposit Date Apr 23, 2015
Publicly Available Date Mar 28, 2024
Journal International Journal of Social Research Methodology
Print ISSN 1364-5579
Electronic ISSN 1464-5300
Publisher Taylor and Francis Group
Peer Reviewed Peer Reviewed
Volume 19
Issue 4
Pages 481-490
DOI https://doi.org/10.1080/13645579.2015.1091235
Keywords Significance testing, Sampling, Attrition, Evaluation, Reliability.

Files

Accepted Journal Article (325 Kb)
PDF

Copyright Statement
© 2015 The Author(s). Published by Taylor & Francis. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.



Published Journal Article (Final published version) (429 Kb)
PDF

Copyright Statement
Final published version




You might also like



Downloadable Citations