Tymms, P. and Higgins, S. (2018) 'Judging research papers for research excellence.', Studies in higher education., 43 (9). pp. 1548-1560.
The UK’s Research Excellence Framework of 2014 was an expensive high stakes evaluation which had a range of impacts on higher education institutions across the country. One component was an assessment of the quality of research outputs where a major feature was a series of panels organised to read and rate the outputs of their peers. Quality control was strengthened after the Research Assessment Exercise of 2008, but questions still remain about how fair it is to rate all papers on the same scale by raters who may vary in both their reliability and their severity/leniency. This paper takes data from a large department in which 23 senior staff rated the outputs from 42 academics. In total, 710 ratings were recorded. The analyses, using the Rasch model, showed that: a single scale described the data well; most raters were reliable although two were idiosyncratic; there was, however, a noticeable variation in the severity/leniency of the raters, which should be taken into account in the overall assessment. Suggestions for future exercises include a pre-appointment procedure for panel members and statistical adjustments for the severity/leniency of raters.
|Full text:||(AM) Accepted Manuscript|
First Live Deposit - 24 November 2016
Download PDF (1008Kb)
|Publisher Web site:||https://doi.org/10.1080/03075079.2016.1266609|
|Publisher statement:||This is an Accepted Manuscript of an article published by Taylor & Francis Group in Studies in higher education on 30/01/2017, available online at: http://www.tandfonline.com/10.1080/03075079.2016.1266609.|
|Record Created:||24 Nov 2016 10:53|
|Last Modified:||21 Aug 2018 11:31|
|Social bookmarking:||Export: EndNote, Zotero | BibTex|
|Look up in GoogleScholar | Find in a UK Library|