We use cookies to ensure that we give you the best experience on our website. By continuing to browse this repository, you give consent for essential cookies to be used. You can read more about our Privacy and Cookie Policy.

Durham Research Online
You are in:

Trends in the quality of human-centric software engineering experiments--a quasi-experiment.

Kitchenham, Barbara and Sjoberg, Dag and Dyba, Tore and Brereton, Pearl and Budgen, David and Host, Martin and Runeson, Per (2013) 'Trends in the quality of human-centric software engineering experiments--a quasi-experiment.', IEEE Transactions on Software Engineering, 39 (7). 1002-1017 .


Context: Several text books and articles published between 2000 and 2002 have attempted to introduce experimental design and statistical methods to software engineers undertaking empirical studies. Objective: This paper investigates whether there has been an increase in the quality of human-centric experimental and quasi-experimental journal papers over the time period 1993 to 2010. Method: 70 experimental and quasi-experimental papers published in four general software engineering journals in the years 1992-2002 and 2006-2010 were each assessed for quality by three empirical software engineering researchers using two quality assessment methods (a questionnaire based method and a subjective overall assessment). Regression analysis was used to assess the relationship between paper quality and the year of publication, publication date group (before 2003 and after 2005), source journal, average co-author experience, citation of statistical text books and articles, and paper length. The results were validated both by removing papers for which the quality score appeared unreliable and using an alternative quality measure. Results: Paper quality was significantly associated with year, citing general statistical texts and paper length (p<0.05). Paper length did not reach significance when quality was measured using an overall subjective assessment. Conclusions: The quality of experimental and quasi-experimental software engineering papers appears to have improved gradually since 1993.

Item Type:Article
Keywords:Codes of good practice, Computing milieux, Miscellaneous, Software engineering, Software/software engineering, The computing profession.
Full text:(AM) Accepted Manuscript
Download PDF
Publisher Web site:
Publisher statement:© 2012 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.”
Date accepted:No date available
Date deposited:14 December 2012
Date of first online publication:July 2013
Date first made open access:No date available

Save or Share this output

Look up in GoogleScholar