We use cookies to ensure that we give you the best experience on our website. By continuing to browse this repository, you give consent for essential cookies to be used. You can read more about our Privacy and Cookie Policy.

Durham Research Online
You are in:

What to do instead of significance testing? Calculating the ‘number of counterfactual cases needed to disturb a finding’.

Gorard, S. and Gorard, J. (2016) 'What to do instead of significance testing? Calculating the ‘number of counterfactual cases needed to disturb a finding’.', International journal of social research methodology., 19 (4). pp. 481-490.


This brief paper introduces a new approach to assessing the trustworthiness of research comparisons when expressed numerically. The ‘number needed to disturb’ a research finding would be the number of counterfactual values that can be added to the smallest arm of any comparison before the difference or ‘effect’ size disappears, minus the number of cases missing key values. This way of presenting the security of findings has several advantages over the use of significance tests, effect sizes and confidence intervals. It is not predicated on random sampling, full response or any specific distribution of data. It bundles together the sample size, magnitude of the finding and the level of attrition in a way that is standardised and therefore comparable between studies.

Item Type:Article
Keywords:Significance testing, Sampling, Attrition, Evaluation, Reliability.
Full text:(AM) Accepted Manuscript
Download PDF
Full text:(VoR) Version of Record
Available under License - Creative Commons Attribution.
Download PDF (Advance online version)
Full text:(VoR) Version of Record
Download PDF (Final published version)
Publisher Web site:
Publisher statement:© 2015 The Author(s). Published by Taylor & Francis. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Date accepted:23 August 2015
Date deposited:30 April 2015
Date of first online publication:06 October 2015
Date first made open access:No date available

Save or Share this output

Look up in GoogleScholar