Raters’ Judgments Reliability in the Spotlight: Assessment Challenges in Pronunciation Research within ESP Teaching

Abstract

Many researchers nowadays study pronunciation as a means to improve second language students’ communicative skills. Native listeners’ perception tasks are considered a reliable tool although their veracity has been questioned due to the listeners’ subjectivity and/or the type of task performed. The present study investigates the effectiveness of explicit rhythm instruction to Spanish/Catalan bilinguals who were engineering undergraduates to improve their comprehensibility and fluency in English. For this purpose, seven native speakers of English rated the extemporaneous speeches of forty-two students before and after instruction via a 5-point scale. Students were divided into an experimental group, receiving explicit rhythm instruction, and a control group, which did not. Unexpectedly, results showed that the experimental group was less comprehensible after treatment, and that high-level learners of both groups performed worse. After close examination, a ceiling effect was detected, caused by an overrating of students before training. Hence, the use of a mixed-method assessment is encouraged to support human judgments.