What Shapes Automated Ratings in Computer-Based English Speaking Tests? Perspectives from Analytic Complexity, Accuracy, Fluency, and Pronunciation Indices

Abstract

The computer-based English speaking test (CBEST) represents a significant advancement in assessing oral proficiency through automated scoring. However, its scoring validity and alignment with well-established linguistic proficiency measures remain underexplored. This study examines the relationship between automated ratings (ARs) and analytic complexity, accuracy, fluency, and pronunciation (CAFP) indices in the context of Zhongkao, namely China’s Senior High School Entrance Examination. Using a quantitative approach, we analyzed correlations between ARs and 12 CAFP indices from a random sample of 382 mock CBEST recordings from an officially recognized platform. Findings revealed strong correlations between ARs and speaking accuracy, fluency, and intelligibility, with fluency and intelligibility significantly predicting ARs. However, no correlations were found with complexity, comprehensibility, or accentedness. Further analysis within CAFP primarily demonstrated correlations between syntactic and lexical complexity with accuracy, accuracy with fluency (the number of repairs and speech rate) and intelligibility, and fluency (speech rate) with intelligibility. However, certain indices such as comprehensibility and accentedness showed no correlation with others. These findings suggest that CBEST prioritizes certain aspects of speaking proficiency, potentially overlooking others. Implications highlight the need for teachers and test designers to align instruction with test expectations while refining CBEST scoring mechanisms to enhance validity and comprehensiveness.