Hype or hope? A meta-analysis of conversational chatbots on L2 learning
(2026) EUROCALL 2026 — Location: Ulster University, Belfast (8.September.2026)
Files
No attached file found for this publication.
Details
- Authors
- Abstract
- (en) Dialogue-based computer-assisted language learning (CALL) can be understood as "meaningful conversational interactions" with "automated agents" (Bibauw et al., 2019, p. 827) for second language (L2) learning. Grounded in the interactionist approach to SLA, it offers learners unique opportunities for "pushed output" in authentic contexts. The emergence of Generative AI tools since 2022 has revolutionised this field by dramatically broadening access to chatbots and enhancing the naturalness and pedagogical potential of interactions, warranting a revisit to the conclusions of previous meta-analyses on dialogue-based CALL (i.e., Bibauw et al., 2022; Hou & Min, 2025). These earlier works did not compare Generative AI with previous generations of chatbots, nor did they examine the conditions (e.g., human interlocutors, traditional classroom instruction) against which chatbots were found to be effective. Therefore, the proposed meta-analysis attempts to synthesize evidence on the effects of meaningful interaction with bots (e.g., chatbots, robots, and non-player agents in XR environments) on language learning outcomes. Through an exhaustive search of multiple databases in both English and Chinese that incorporated previously overlooked terms related to robots and extended reality, 67 studies published between 2011 and 2025 were retained, yielding 237 effect sizes based on 4,544 participants across 23 L1 backgrounds. From this pool, we reconceptualize dialogue-based CALL as a continuum and propose a typology based on the freedom in learners' spontaneous output, ranging from non-verbal responses, repetition, limited verbal output, to free speech, offering a simplified and practical alternative to the earlier framework (i.e., Bibauw et al., 2019) that would remain valid amid the evolving technological landscape. A three-level random-effects model yielded a positive, medium-sized overall effect (g = 0.61, 95% CI [0.48, 0.74], p < .001), confirming that meaningful interaction with dialogue systems improves L2 performance. Moderator analyses revealed that age group significantly influenced outcomes, with teenagers showing the largest gains (g = 1.05), followed by adults (g = 0.54) and children (g = 0.49). Outcome type also emerged as a significant moderator: dialogue systems were most beneficial for productive skills (g = 0.65) and language knowledge (g = 0.60)—particularly vocabulary (g = 0.92)—while their impact on comprehension remained non-significant. A notable finding was that matched test–practice modality conditions produced effect sizes approximately 2.5 times larger than mismatched ones, and delayed posttests (g = 0.92) outperformed immediate ones (g = 0.58), suggesting sustained learning benefits. While the underlying technology was not a significant moderator, recent LLM-powered chatbots showed the largest estimates (g = 0.64). Positive effects were observed across both in-class and out-of-class settings. In sum, this study contributes to updating the conceptual framework of dialogue-based CALL, facilitating the transfer of empirical results into real practices through more consistent terminology, proposing an evaluation framework of chatbot implementations and guidelines for future effectiveness studies, and indicating directions for the design of chatbots and pedagogical strategies.
- Affiliations
Citations
Wang, Z., Bibauw, S., Noreillie, A.-S., & Desmet, P. (2026, September 8). Hype or hope? A meta-analysis of conversational chatbots on L2 learning. EUROCALL 2026, Ulster University, Belfast. https://hdl.handle.net/2078.5/278464