Tymms, P. and Higgins, S. (2018) 'Judging research papers for research excellence.', Studies in higher education., 43 (9). pp. 1548-1560.
Abstract
The UK’s Research Excellence Framework of 2014 was an expensive high stakes evaluation which had a range of impacts on higher education institutions across the country. One component was an assessment of the quality of research outputs where a major feature was a series of panels organised to read and rate the outputs of their peers. Quality control was strengthened after the Research Assessment Exercise of 2008, but questions still remain about how fair it is to rate all papers on the same scale by raters who may vary in both their reliability and their severity/leniency. This paper takes data from a large department in which 23 senior staff rated the outputs from 42 academics. In total, 710 ratings were recorded. The analyses, using the Rasch model, showed that: a single scale described the data well; most raters were reliable although two were idiosyncratic; there was, however, a noticeable variation in the severity/leniency of the raters, which should be taken into account in the overall assessment. Suggestions for future exercises include a pre-appointment procedure for panel members and statistical adjustments for the severity/leniency of raters.
Item Type: | Article |
---|---|
Full text: | (AM) Accepted Manuscript Download PDF (1008Kb) |
Status: | Peer-reviewed |
Publisher Web site: | https://doi.org/10.1080/03075079.2016.1266609 |
Publisher statement: | This is an Accepted Manuscript of an article published by Taylor & Francis Group in Studies in higher education on 30/01/2017, available online at: http://www.tandfonline.com/10.1080/03075079.2016.1266609. |
Date accepted: | 21 November 2016 |
Date deposited: | 24 November 2016 |
Date of first online publication: | 30 January 2017 |
Date first made open access: | 30 July 2018 |
Save or Share this output
Export: | |
Look up in GoogleScholar |