We use cookies to ensure that we give you the best experience on our website. By continuing to browse this repository, you give consent for essential cookies to be used. You can read more about our Privacy and Cookie Policy.

Durham Research Online
You are in:

Judging research papers for research excellence.

Tymms, P. and Higgins, S. (2018) 'Judging research papers for research excellence.', Studies in higher education., 43 (9). pp. 1548-1560.


The UK’s Research Excellence Framework of 2014 was an expensive high stakes evaluation which had a range of impacts on higher education institutions across the country. One component was an assessment of the quality of research outputs where a major feature was a series of panels organised to read and rate the outputs of their peers. Quality control was strengthened after the Research Assessment Exercise of 2008, but questions still remain about how fair it is to rate all papers on the same scale by raters who may vary in both their reliability and their severity/leniency. This paper takes data from a large department in which 23 senior staff rated the outputs from 42 academics. In total, 710 ratings were recorded. The analyses, using the Rasch model, showed that: a single scale described the data well; most raters were reliable although two were idiosyncratic; there was, however, a noticeable variation in the severity/leniency of the raters, which should be taken into account in the overall assessment. Suggestions for future exercises include a pre-appointment procedure for panel members and statistical adjustments for the severity/leniency of raters.

Item Type:Article
Full text:(AM) Accepted Manuscript
Download PDF
Publisher Web site:
Publisher statement:This is an Accepted Manuscript of an article published by Taylor & Francis Group in Studies in higher education on 30/01/2017, available online at:
Date accepted:21 November 2016
Date deposited:24 November 2016
Date of first online publication:30 January 2017
Date first made open access:30 July 2018

Save or Share this output

Look up in GoogleScholar