Comparative judgement for L2 writing assessment: Do narrow proficiency range and homogeneous topics affect reliability?
(2023) The 32nd Conference of the European Second Language Association (EuroSLA32) — Location: Birmingham, UK (30.August.2023)
Files
No attached file found for this publication.
Details
- Authors
- Abstract
- Comparative Judgement (CJ) is a method of assessment in which judges (who may be experts, peers or even novices) decide which of two pieces of student work is “better”. These comparisons can be used to automatically rank each piece of work, and from these rankings a grade can be derived. Research from a range of fields has demonstrated that CJ generates reliable, valid, and efficient evaluations of learner outputs (Jones et al., 2019; Wheadon et al., 2020). This presentation reports on findings from a project which explores the potential for CJ (in tandem with crowdsourcing methods) to contribute to the assessment of L2 texts included in widely used learner corpora that unfortunately often do not come with reliable proficiency measurement. A recently-published preliminary study (Author, 2022) suggested CJ’s suitability for this purpose. However, in focusing on relatively short writing samples (median = 272 words), using a single essay prompt and covering the full spectrum of L2 proficiency, the study provided optimal conditions for such results. A next step is to find out whether CJ continues to be reliable when learner texts are longer, more homogeneous, and/or include responses to a range of prompts. Since all of these changes are likely to complicate the task of choosing the stronger of each pair of essays (van Daal et al., 2019), it is important to test their effects on the decisions made by judges. The two studies reported here therefore involve the comparative judgement of two sets of texts; firstly 48 learner essays taken from the International Corpus of Learner English, constituting responses to a single prompt, from a relatively narrow proficiency range (roughly B1-C1), and of greater length than the texts in the earlier study (median = 548 words); and secondly another set of texts, similar except that they comprised responses to five different topics. In addition to being evaluated using CJ, all texts were triple-graded using rubric-based methods. The presentation will compare the reliability and efficiency of these two assessment approaches. The results contribute both to learner corpus research, where CJ can potentially enrich learner corpora with accurate measures of text proficiency; and to the more general goal of exploring reliable, valid, and efficient alternatives to rubric-based L2 writing assessment.
- Affiliations
Citations
Thwaites, P., Kollias, C., & Paquot, M. (2023). Comparative judgement for L2 writing assessment: Do narrow proficiency range and homogeneous topics affect reliability? The 32nd Conference of the European Second Language Association (EuroSLA32), Birmingham, UK. https://hdl.handle.net/2078.5/269000