Crowdsourced comparative judgement of L2 writing: Does a narrow proficiency range reduce test reliability?
(2023) ALTE 8th International Conference — Location: Madrid (26.April.2023)
Files
No attached file found for this publication.
Details
- Authors
- et. al.
- Abstract
- Comparative Judgement (CJ) is a method of assessment in which judges (who may be experts, peers or even novices) decide which of two pieces of student work is “better”. These comparisons can be used to automatically rank each piece of work, and from these rankings a grade can be derived. Research from a range of fields has demonstrated that CJ generates reliable, valid, and efficient evaluations of learner outputs (e.g. Jones et al., 2019; Wheadon, 2022). This presentation reports on early findings from a project exploring the potential for CJ (in tandem with crowdsourcing methods) to contribute to the assessment of L2 texts included in widely used learner corpora that unfortunately often do not come with reliable proficiency measurement. A preliminary study suggested CJ’s reliability for this purpose. However, in focusing on writing samples covering the full spectrum of L2 proficiency (see also Sims et al., 2020), the study provided optimal conditions for such results. A next step is to find out whether CJ continues to be reliable when learner texts are more homogeneous, and therefore more difficult for judges to compare (van Daal et al., 2019). The study reported here therefore involves the comparative judgement of texts from a narrower proficiency range (roughly B1-C1) from the International Corpus of Learner English (Granger et al, 2020). In addition to exploring the reliability of these evaluations, the paper also offers a first attempt at exploring the validity of CJ for L2 assessment by testing which aspects of learner texts (e.g. length, lexical sophistication etc.) best predict CJ-derived evaluations of text quality. The results contribute both to learner corpus research, where CJ can potentially enrich learner corpora with accurate measures of text proficiency; and to the more general goal of exploring reliable, valid, and efficient alternatives to rubric-based L2 assessment.
- Affiliations
Citations
Thwaites, P., Paquot, M., & et al. (2023). Crowdsourced comparative judgement of L2 writing: Does a narrow proficiency range reduce test reliability? ALTE 8th International Conference, Madrid. https://hdl.handle.net/2078.5/269249