Comparative judgement for L2 writing assessment: are expert judges necessary?
(2023) EALTA Joint Special Interest Group Meeting for the Assessment of Writing and Academic Purposes — Location: Helsinki (15.June.2023)
Files
No attached file found for this publication.
Details
- Authors
Vandeweerd, Nathan 
- et. al.
- Abstract
- Comparative Judgement (CJ) is a method of assessment in which judges decide which of two productions is “better”. Research from numerous fields has demonstrated that CJ generates reliable, valid, and efficient evaluations (e.g. Jones et al., 2019). Recent studies have suggested that these benefits might also apply to the use of CJ for L2 writing assessment (Şahin, 2021; Sims et al., 2020). An unresolved question in CJ research concerns the level of expertise judges require to generate accurate and reliable assessments (Bartholomew & Jones, 2022). This presentation reports on a study exploring this question by combining CJ with crowdsourcing methods. In the first part of the study (Author, 2022), 50 L2 English academicargumentative essays from the ETS corpus were judged by members of the linguistic community. In an ongoing second stage, the same set of essays was judged by a more diverse group of judges recruited through Prolific, an online crowdsourcing platform. The key difference between the two groups is therefore their level of linguistic expertise. In the presentation, the ratings scales generated by the two groups will be compared: their reliability, their extent of agreement, and their correlations with rubric-based ETS scores. The results of the study will be of primary interest to the field of learner corpus research, where crowdsourced CJ offers potential for reliable text assessment (Paquot et al., 2022). More generally, by exploring the role of judge expertise they will contribute to discussions regarding the feasibility of CJ for second language academic writing assessment.
- Affiliations
Citations
Thwaites, P., Vandeweerd, N., Paquot, M., & et al. (2023). Comparative judgement for L2 writing assessment: are expert judges necessary? EALTA Joint Special Interest Group Meeting for the Assessment of Writing and Academic Purposes, Helsinki. https://hdl.handle.net/2078.5/268990