Exploring the Construct of Phraseological Competence: Perspectives from Human Raters Through Comparative Judgment
(2025) Linguists Day of the Linguistic Society of Belgium (LSB) 2025 — Location: KU Leuven (10.October.2025)
Files
Abstract_Final_version_Tingting_LinguistsDayLSB.docx
Open Access - Microsoft Word XML
- 29.65 KB
Details
- Authors
- Abstract
- Phraseological competence is crucial for language acquisition, processing, and fluency (Ellis et al., 2008; Paquot et al., 2020) but remains challenging for L2 learners (Laufer & Waldman, 2011; Paquot & Granger, 2012). Recent empirical studies have increasingly conceptualized phraseological competence as a multidimensional construct that includes facets such as accuracy, depth, and breadth (Naismith & Juffs, 2025; Paquot, 2019; Xu, 2018). This conceptualization raises important questions about how phraseological competence should be assessed and which dimensions are most salient in different contexts. In research on linguistic constructs more broadly, two complementary approaches are commonly adopted: (1) analyzing learner output to identify salient performance features, and (2) examining rater orientations to uncover the criteria underlying expert judgments (Ducasse & Brown, 2009). Within phraseological research, however, the first approach—relying on corpus-based analyses and computational indices—has been overwhelmingly dominant (e.g., Paquot, 2019; Paquot & Naets, 2025). In contrast, little is known about the dimensions that human raters focus on when evaluating phraseological competence, even though their judgments are often regarded as the gold standard for establishing construct validity (Crossley et al., 2013). This study investigates how human raters assess phraseological competence, with the aim of identifying the specific dimensions to which they are most sensitive. To this end, it adopts comparative judgment (CJ), a methodological innovation increasingly used to assess complex and multidimensional linguistic constructs in applied linguistics (Thwaites & Paquot, 2024; Verhavert et al., 2019). In CJ, expert raters compare pairs of performances and decide which one demonstrates a higher level of the target construct. These pairwise comparisons are then modelled statistically using the Bradley-Terry-Luce (BTL) framework. CJ has proven to be a reliable and valid method for assessing complex constructs in language learning and assessment (Thwaites et al., 2024; Verhavert et al., 2019). The study addresses two questions: RQ1. How reliable are the rating scales of test takers’ phraseological competence generated through human comparative judgements? RQ2. What dimensions of phraseological competence do raters attend to when using CJ methods for assessment? To investigate these questions, the study involved 30 doctoral students and university instructors in Applied Linguistics who served as judges. Each judge completed 49 pairwise comparisons based on transcripts of oral performances from 98 candidates who sat the Test for English Majors Band 8 Oral (TEM 8-Oral) exam. For every comparison, judges also provided written justifications for their decision. The reliability of the resulting scale was measured using Scale Separation Reliability (SSR). In addition, a thematic analysis of the qualitative comments was conducted to identify the specific aspects of phraseological competence (e.g., accuracy, diversity, sophistication) that featured most prominently in human judgments. After all comparisons were completed, the CJ-generated rating scales demonstrated high reliability, with SSR reaching 0.88. Thematic coding of 953 justifications from the judges (approximately 20% of all data) revealed that raters attended to multiple dimensions of phraseological competence, including accuracy, appropriateness, phraseological sophistication, diversity, idiomaticity, quantity, and fluency. Among these, accuracy was the most frequently emphasized (37.04%), followed by phraseological sophistication (19.80%), diversity (16.57%), appropriateness (12.49%), idiomaticity (7.99%), quantity (5.52%), and fluency (0.59%). These findings shed light on the dimensions foregrounded by raters, provide empirical support for the multidimensional nature of phraseological competence, and demonstrate the value of CJ as a tool for construct exploration in language assessment. We are currently conducting double annotation for 10% of the dataset to further establish inter-coder reliability. Results from the full dataset will be presented at the conference. References Crossley, S., Salsbury, T., & McNamara, D. S. (2013). Validating lexical measures using human scores of lexical proficiency. In S. Jarvis & M. Daller (Eds.), Studies in bilingualism (Vol. 47, pp. 105-134). John Benjamins Publishing Company. https://doi.org/10.1075/sibil.47.06ch4 Ducasse, A. M., & Brown, A. (2009). Assessing paired orals: Raters’ orientation to interaction. Language Testing, 26(3), 423–443. https://doi.org/10.1177/0265532209104669 Ellis, N. C., Simpson-Vlach, R., & Maynard, C. (2008). Formulaic language in native and second language speakers: Psycholinguistics, corpus linguistics, and TESOL. TESOL Quarterly, 42(3), 375–396. https://doi.org/10.1002/j.1545-7249.2008.tb00137.x Laufer, B., & Waldman, T. (2011). Verb-noun collocations in second language writing: A corpus analysis of learners’ English. Language Learning, 61(2), 647–672. https://doi.org/10.1111/j.1467-9922.2010.00621.x Naismith, B., & Juffs, A. (2025). The impact of collocational proficiency features on expert ratings of L2 English learners’ writing. Studies in Second Language Acquisition, 1–25. https://doi.org/10.1017/S0272263125000075 Paquot, M., & Granger, S. (2012). Formulaic language in learner corpora. Annual Review of Applied Linguistics, 32, 130–149. https://doi.org/10.1017/S0267190512000098 Paquot, M., & Naets, H. (2025). Phraseological sophistication as a multidimensional construct: Exploring the relationship between association, register specificity and frequency of word combinations. In T. Larsson & D. Biber (Eds.), Cumulative knowledge building and replication in Learner Corpus Research. International Journal of Learner Corpus Research, 11(1). https://doi.org/10.1075/ijlcr.23033.paq Paquot, M., Gries, S. Th., & Yoder, M. (2020). Measuring lexicogrammar. In P. Winke & T. Brunfaut (Eds.), The Routledge handbook of second language acquisition and language testing (pp. 223–232). Routledge. https://doi.org/10.4324/9781351034784-25 Thwaites, P. (2024). Comparative judgement for advancing research in applied linguistics. https://doi.org/10.1016/j.rmal.2024.100142 Verhavert, S., Bouwer, R., Donche, V., & De Maeyer, S. (2019). A meta-analysis on the reliability of comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(5), 541–562. https://doi.org/10.1080/0969594X.2019.1602027 Xu, J. (2018). Measuring “Spoken Collocational Competence” in Communicative Speaking Assessment. Language Assessment Quarterly, 15(3), 255–272. https://doi.org/10.1080/15434303.2018.1482900
- Affiliations
Citations
Wang, T., Paquot, M., & Zhou, D. (2025). Exploring the Construct of Phraseological Competence: Perspectives from Human Raters Through Comparative Judgment. Linguists Day of the Linguistic Society of Belgium (LSB) 2025, KU Leuven.