Bringing the Law of Comparative Judgment into the methodological toolbox of learner corpus researchers
(2023) Graduate Student Conference in Learner Corpus Research — Location: online (25.October.2023)
Files
Abstract.docx
Open Access - Microsoft Word XML
- 32.74 KB
Details
- Authors
- Abstract
- Thurstone’s (1927) Law of Comparative Judgment (CJ) asserts that “a person perceiving some phenomenon will assign to it some instantaneous ‘value’ […], and that when we ask them to choose the ‘better’ of two such phenomena it is these values that are compared” (Pollitt, 2012a:159). This has been reformulated as the assumption that people are able to compare two performances more easily and reliably than to assign a score to an individual performance (Lesterhuis et al., 2017). The CJ technique relies on the consensus of a panel of judges who are asked to compare two performances (e.g. geography essays, academic writing, mathematical problem solving exercises). From many such pairwise comparisons, a scaled distribution of performances is then produced. CJ has developed into a popular methodology in various domains of research and practice, and most particularly in educational assessment, where it is increasingly being valued for “making reliable the assessment of skills that are currently problematic” (Pollit, 2012b: 292). This includes the assessment of writing quality, a multidimensional and complex construct for which CJ tasks have already been reported to produce reliable and valid results (e.g., Lesterhuis et al., 2017; Lesterhuis et al., 2018; van Daal et al., 2019). The main objective of this talk is to introduce the method of CJ to the LCR community and demonstrate its usefulness for enriching L2 data with proficiency scores that are often absent from, or measured in unreliable ways in currently available learner corpora. I will report on a number of studies that we have conducted within the framework of the Crowdsourcing Language Assessment Project (CLAP). Collectively, these studies investigate whether a crowd of judges is able to assess learner texts in a CJ task with high reliability (Paquot et al., 2022; Thwaites & Paquot, 2023; Thwaites et al., in preparation). More specifically, they seek to answer the following research questions: • RQ1. To what extent do specific characteristics of learner texts (topic, length, homogeneity in terms of proficiency) have an effect on the reliability of an ACJ task? • RQ2. To what extent do specific characteristics of judges (language assessment training and expertise) have an effect on the reliability of an ACJ task? • RQ3. To what extent do specific characteristics of learner texts and characteristics of judges have an effect on the validity of an ACJ task? So far, we have only used CJ to enrich learner corpora with proficiency scores but as I will argue by the end of my presentation, CJ may also serve to elicit scores and rankings of productions for other multidimensional constructs (e.g., complexity, accuracy, fluency) that have proved difficult to define and operationalize in the field of L2 research. References Lesterhuis, M., Verhavert, S., Coertjens, L., Donche, V., & De Maeyer, S. (2017). Comparative judgement as a promising alternative to score competences. In E. Cano & G. Ion (Eds.), Innovative practices for higher education assessment and measurement (pp. 119–138). Hershey, PA: IGI Global. Lesterhuis, M., van Daal, T., Van Gasse, R., Coertjens, L., Donche, V., & De Maeyer, S. (2018). When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality. L1 Educational Studies in Language and Literature, 18, 1–22. https://doi.org/10.17239/L1ESLL-2018.18.01.02 Paquot, M., Rubin, R. & Vandeweerd, N. (2022). Crowdsourced Adaptive Comparative Judgment: A community-based solution for proficiency rating. Language Learning, 72(3), 53-885. https://onlinelibrary.wiley.com/doi/10.1111/lang.12498 Pollitt, A. (2012a). Comparative judgement for assessment. International Journal of Technology and Design Education, 22, 157–170. https://doi.org/10.1007/s10798-011-9189-x Pollitt, A. (2012b). The method of adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice, 19, 281–300. https://doi.org/10.1080/0969594X.2012.665354 Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273–86. Thwaites, P. & Paquot, M. (2023). Comparative judgement of L2 writing in learner corpora: does a narrow proficiency range reduce test reliability? Paper to be presented at the ALTE 8th International Conference, 26-28 April 2023, Madrid, Spain. Thwaites, P., Vandeweerd, N., & Paquot, M. (in preparation). Crowdsourcing language assessment: how reliable are laypeople's assessments of L2 writing? van Daal, T., Lesterhuis, M., Coertjens, L., Donche, V., & De Maeyer, S. (2019). Validity of comparative judgement to assess academic writing: Examining implications of its holistic character and building on a shared consensus. Assessment in Education: Principles, Policy & Practice, 26, 59–74. https://doi.org/10.1080/0969594X.2016.1253542
- Affiliations
Citations
Paquot, M. (2023). Bringing the Law of Comparative Judgment into the methodological toolbox of learner corpus researchers. Graduate Student Conference in Learner Corpus Research, online. https://hdl.handle.net/2078.5/104616