Comparability of difficulty levels of translation tasks in CET-6 parallel test forms: evidence from product and process-based data

Abstract

This study investigates the comparability of three parallel translation tasks selected from a College English Test Band-6 (CET-6) and explores the major linguistic features contributing to translation difficulty. Data obtained from the participants’ subjective rating, eye-tracking, and performance evaluation were triangulated to measure the comparability of difficulty levels of parallel translation tasks. Data of word translation entropy, translation errors, and participants’ retrospective reports were correlated to examine the difficulty triggers. The results show that: (i) the text comparability was evidenced by eye-tracking indicators and performance measurements, but not supported by subjective ratings; (ii) the domain content words (DCWs) were reported by the participants as the major cause of translation difficulties and the unequal number of DCWs among the three tasks led to inconsistent ratings for the task difficulty. Our findings suggest that test-takers’ subjective perception and their cognitive skills deserve serious consideration by test designers, as these two factors can better demonstrate difficulty levels among parallel tasks. Our study postulates a new direction to establish a relationship between task characteristics and test validity, and provides suggestions for the CET-6 committee and other examination boards with practical methods to be able to compare the difficulty levels of parallel translation tasks.