Crowdsourced Adaptive Comparative Judgment: A community-based solution for proficiency rating
(2022) Language Learning — Vol. 72, n° 3 (2022), p. 853-885 (2022)
Files
LL_article_revised_DEC2021_FINAL.docx
final version submitted before acceptance
Open Access - Microsoft Word XML
- 445.21 KB
Details
- Authors
- Abstract
- The main objective of this Method Showcase Article is to show how the technique of adaptive comparative judgement, coupled with a crowdsourcing approach can offer practical solutions to the reliability issues as well as the time and cost difficulties associated with a text-based approach to proficiency assessment in L2 research. We showcase this method by reporting on the methodological framework implemented in the CLAP project and presenting the results of a first study that demonstrate that a crowd is able to assess learner texts with high reliability. No effect of language skills or language assessment experience was found on the assessment task, but there seems to be a difference in the decisions made by judges who received formal language assessment training and those who did not. Put together, however, the scores generated by the crowdsourced task exhibit a strong correlation with the rubric-based scores released with the learner corpus used.
- Affiliations
Citations
Paquot, M., Rubin, R., & Vandeweerd, N. (2022). Crowdsourced Adaptive Comparative Judgment: A community-based solution for proficiency rating. Language Learning, 72(3 (2022)), 853-885. https://hdl.handle.net/2078.5/241470 (Original work published 2022)