Abstract
While empirical investigations of artificial intelligence (AI) chatbots for English-as-a-foreign-language (EFL) learners’ language skill development have been receiving considerable attention, a quantitative analysis of its effectiveness and how the pedagogical effects differ under different conditions is still lacking. To bridge the gaps, drawing on the usefulness theoretical framework and the proposed criteria, this study meta-analyzed 21 eligible studies published during 2008–2023 to gain a comprehensive understanding of the overall effect and moderator analyses of AI chatbots for EFL learning. Results indicated that the overall effect was g = 0.648, 95% CI [0.406, 0.891], suggesting that AI chatbots could be effective to facilitate EFL learners’ language skill development. Furthermore, intervention durations and interface designs were found to be significant moderators. Informed by the results obtained, implications for practice were also discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Articles marked with an asterisk (*) were analysed in the meta-analysis
Bahari, A. (2022). Affordances and challenges of technology-assisted language learning for motivation: A systematic review. Interactive Learning Environments. https://doi.org/10.1080/10494820.2021.2021246
*Çakmak, F. (2022). Chatbot-human interaction and its effects on EFL students’ L2 speaking performance and anxiety. Novitas-ROYAL, 16(2), 113–131. https://doi.org/10.3969/j.issn.1307-4733.2022.16.002
Chen, C. M., & Lin, Y. J. (2016). Effects of different text display types on reading comprehension, sustained attention and cognitive load in mobile reading contexts. Interactive Learning Environments, 24(3), 553–571. https://doi.org/10.1080/10494820.2014.891526
Chen, Z., Chen, W., Jia, J., & An, H. (2020). The effects of using mobile devices on language learning: A meta-analysis. Educational Technology Research and Development, 68(4), 1769–1789. https://doi.org/10.1007/s11423-020-09801-5
Chin, K. Y., Hong, Z. W., & Chen, Y. L. (2014). Impact of using an educational robot-based learning system on students’ motivation in elementary education. IEEE Transactions on Learning Technologies, 7(4), 333–345. https://doi.org/10.1109/TLT.2014.2346756
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum. https://doi.org/10.1080/00401706.1989.10488618
*Dizon, G. (2020). Evaluating intelligent personal assistants for L2 listening and speaking development. Language Learning & Technology, 24(1), 16–26. https://www.lltjournal.org/item/934/
*El Shazly, R. (2021). Effects of artificial intelligence on English speaking anxiety and speaking performance: A case study. Expert Systems, 38(3), 1–15. https://doi.org/10.1111/exsy.12667
Fidan, M., & Gencel, N. (2022). Supporting the instructional videos with chatbot and peer feedback mechanisms in online learning: The effects on learning performance and intrinsic motivation. Journal of Educational Computing Research, 60(7), 1716–1741. https://doi.org/10.1177/07356331221077901
Fryer, L. K., Nakao, K., & Thompson, A. (2019). Chatbot learning partners: Connecting learning experiences, interest and competence. Computers in Human Behavior, 93, 279–289. https://doi.org/10.1016/j.chb.2018.12.023
Gibson, J. J. (1977). The theory of affordances. In R. Shaw & J. Bransford (Eds.), Perceiving, acting, and knowing: Toward an ecological psychology (pp. 67–82). Erlbaum.
Goda, Y., Yamada, M., Matsukawa, H., Hata, K., & Yasunami, S. (2014). Conversation with a chatbot before an online EFL group discussion and the effects on critical thinking. The Journal of Information and Systems in Education, 13(1), 1–7. https://doi.org/10.12937/ejsise.13.1
*Hadi, M. S., & Junor, R. S. (2022). Speaking to devices: Can we use Google Assistant to foster students’ speaking skills?. Journal of Languages and Language Teaching, 10(4), 570–578. https://doi.org/10.33394/jollt.v10i4.5808
*Han, D. E. (2020). The Effects of voice-based AI chatbots on Korean EFL middle school students’ speaking competence and affective domains. Asia-Pacific Journal of Convergent Research Interchange, 6(7), 71–80. https://doi.org/10.47116/apjcri.2020.07.07
Higgins, K., Huscroft-D’Angelo, J., & Crawford, L. (2019). Effects of technology in mathematics on achievement, motivation, and attitude: A meta-analysis. Journal of Educational Computing Research, 57(2), 283–319. https://doi.org/10.1177/0735633117748416
Hsu, H. L., Chen, H. H. J., & Todd, A. G. (2021). Investigating the impact of the Amazon Alexa on the development of L2 listening and speaking skills. Interactive Learning Environments. https://doi.org/10.1080/10494820.2021.2016864
Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning—Are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38(1), 237–257. https://doi.org/10.1111/jcal.12610
*Jensen, C. J. D., & Cadierno, T. (2022). Differences in mobile-assisted acquisition of receptive and productive vocabulary knowledge: A case study using Mondly. The Language Learning Journal. https://doi.org/10.1080/09571736.2022.2108123
*Jeon, J. (2021). Chatbot-assisted dynamic assessment (CA-DA) for L2 vocabulary learning and diagnosis. Computer Assisted Language Learning. https://doi.org/10.1080/09588221.2021.1987272
Ji, H., Han, I., & Ko, Y. (2023). A systematic review of conversational AI in language education: Focusing on the collaboration with human teachers. Journal of Research on Technology in Education, 55(1), 48–63. https://doi.org/10.1080/15391523.2022.2142873
*Jia, J., Chen, Y., Ding, Z., & Ruan, M. (2012). Effects of a vocabulary acquisition and assessment system on students’ performance in a blended learning class for English subject. Computers & Education, 58(1), 63–76. https://doi.org/10.1016/j.compedu.2011.08.002
*Jia, J., & Ruan, M. (2008). Use chatbot CSIEC to facilitate the individual learning in English instruction: A case study. In Intelligent tutoring systems: 9th international conference, ITS 2008, Montreal, Canada, June 23–27, 2008 proceedings 9 (pp. 706–708). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-69132-7_84
*Junaidi, J. (2020). Artificial intelligence in EFL context: Rising students’ speaking performance with Lyra virtual assistance. International Journal of Advanced Science and Technology Rehabilitation, 29(5), 6735–6741. https://doi.org/10.3969/j.issn.2005-4238.2020.29.005
*Kang, H. (2022). Effects of artificial intelligence (AI) and native speaker interlocutors on ESL learners’ speaking ability and affective aspects. Multimedia-Assisted Language Learning, 25(2), 9–43. https://doi.org/10.15702/mall.2022.25.2.09
*Kim, H. S., Cha, Y., & Kim, N. Y. (2021). Effects of AI chatbots on EFL students’ communication skills. Korean Journal of English Language and Linguistics, 21, 712–734. https://doi.org/10.15738/kjell.21.202108.712
*Kim, N. Y. (2016). Effects of voice chat on EFL learners’ speaking ability according to proficiency levels. Multimedia-Assisted Language Learning, 19(4), 63–88. https://doi.org/10.15702/mall.2016.19.4.63
*Kim, N. Y. (2018a). Chatbots and Korean EFL students’ English vocabulary learning. Journal of Digital Convergence, 16(2), 1–7. https://doi.org/10.14400/JDC.2018.16.2.001
*Kim, N. Y. (2018b). A study on chatbots for develo** Korean college students’ English listening and reading skills. Journal of Digital Convergence, 16(8), 19–26. https://doi.org/10.14400/JDC.2018.16.8.019
*Kim, N. Y. (2019). A study on the use of artificial intelligence chatbots for improving English grammar skills. Journal of Digital Convergence, 17(8), 37–46. https://doi.org/10.14400/JDC.2019.17.8.037
Kirschner, P., Strijbos, J. W., Kreijns, K., & Beers, P. J. (2004). Designing electronic collaborative learning environments. Educational Technology Research and Development, 52(3), 47–66. https://doi.org/10.2307/30220390
Kreijns, K., Kirschner, P. A., & Jochems, W. (2002). The sociability of computer-supported collaborative learning environments. Educational Technology & Society, 5(1), 8–22. https://doi.org/10.3969/j.issn.1436-4522.2002.5.001
Lee, H., & Lee, J. H. (2022). The effects of robot-assisted language learning: A meta-analysis. Educational Research Review, 35, 1–13. https://doi.org/10.1016/j.edurev.2021.100425
Lee, J. Y., & Hwang, Y. (2022). A meta-analysis of the effects of using AI chatbot in Korean EFL education. Studies in English Language & Literature, 48(1), 213–243. https://doi.org/10.21559/AELLK.2022.48.1.011
Lee, S., & Jeon, J. (2022). Visualizing a disembodied agent: Young EFL learners’ perceptions of voice-controlled conversational agents as language partners. Computer Assisted Language Learning. https://doi.org/10.1080/09588221.2022.2067182
Li, R. (2021). Does game-based vocabulary learning APP influence Chinese EFL learners’ vocabulary achievement, motivation and self-confidence? SAGE Open, 11(1), 1–12. https://doi.org/10.1177/21582440211003092
Li, R. (2022a). Effects of mobile-assisted language learning on EFL/ESL reading comprehension. Educational Technology & Society, 25(3), 15–29.
Li, R. (2022b). Effects of blended language learning on EFL learners’ language performance: An activity theory approach. Journal of Computer Assisted Learning, 38, 1273–1285. https://doi.org/10.1111/jcal.12697
Li, R. (2022c). Examining foreign language listening anxiety and its correlates: A meta-analysis. Porta Linguarum, 38, 99–117. https://doi.org/10.30827/portalin.vi38.22173
Li, R. (2022d). Research trends of blended language learning: A bibliometric synthesis of SSCI-indexed journal articles during 2000–2019. ReCALL, 34(3), 309–326. https://doi.org/10.1017/S0958344021000343
Li, R. (2022e). Foreign language reading anxiety and its correlates: A meta-analysis. Reading and Writing, 35(4), 995–1018. https://doi.org/10.1007/s11145-021-10213-x
Li, R. (2023a). Investigating effects of computer-mediated feedback on L2 vocabulary learning. Computers & Education, 198, 1–11. https://doi.org/10.1016/j.compedu.2023.104763
Li, R. (2023b). Effects of mobile-assisted language learning on EFL learners’ listening skill development. Educational Technology & Society, 26(2), 36–49.
Li, R. (2023c). Still a fallible tool? Revisiting effects of automated writing evaluation from activity theory perspective. British Journal of Educational Technology, 54(3), 773–789. https://doi.org/10.1111/bjet.13294
Li, R. (2024). Effects of mobile-assisted language learning on foreign language learners’ speaking skill development. Language Learning & Technology, 28(1), 1–26.
Li, R., Meng, Z., Tian, M., Zhang, Z., & **ao, W. (2021). Modelling Chinese EFL learners’ flow experiences in digital game-based vocabulary learning: The roles of learner and contextual factors. Computer Assisted Language Learning, 34(4), 483–505. https://doi.org/10.1080/09588221.2019.1619585
Lin, V., Yeh, H. C., Huang, H. H., & Chen, N. S. (2022). Enhancing EFL vocabulary learning with multimodal cues supported by an educational robot and an IoT-based 3D book. System, 104, 102–121. https://doi.org/10.1016/j.system.2021.102691
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Sage.
McTear, M. F., Callejas, Z., & Griol, D. (2016). The conversational interface. Springer. https://doi.org/10.1007/978-3-319-32967-32
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151(4), 264–269. https://doi.org/10.7326/0003-819-151-4-200908180-00135
Radziwill, N. M., & Benton, M. C. (2017). Evaluating quality of chatbots and intelligent conversational agents. Software Quality Professional, 19(3), 25–35. https://doi.org/10.48550/ar**v.1704.04579
Rosenthal, R. (1991). Meta-analytic procedures for social research. Sage. https://doi.org/10.3102/0013189X015008018
Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education, 151, 1–11. https://doi.org/10.1016/j.compedu.2020.103862
*Tai, T. Y. (2022). Effects of intelligent personal assistants on EFL learners’ oral proficiency outside the classroom. Computer Assisted Language Learning. https://doi.org/10.1080/09588221.2022.2075013
*Tai, T. Y., & Chen, H. H. J. (2022a). The impact of intelligent personal assistants on adolescent EFL learners’ listening comprehension. Computer Assisted Language Learning. https://doi.org/10.1080/09588221.2022.2040536
*Tai, T. Y., & Chen, H. H. J. (2022b). The impact of intelligent personal assistants on adolescent EFL learners’ speaking proficiency. Computer Assisted Language Learning. https://doi.org/10.1080/09588221.2022.2070219
*Tai, T. Y., Chen, H. H. J., & Todd, G. (2020). The impact of a virtual reality app on adolescent EFL learners’ vocabulary learning. Computer Assisted Language Learning, 35(4), 892–917. https://doi.org/10.1080/09588221.2020.1752735
Weizenbaum, J. (1966). Eliza—A computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36–45. https://doi.org/10.1145/365153.365168
Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M., & Drachsler, H. (2021). Are we there yet?—A systematic literature review on chatbots in education. Frontiers in Artificial Intelligence, 4, 1–18. https://doi.org/10.3389/frai.2021.654924
Yang, L., Li, R., & Zhou, Y. (2024). Research trends of game-based language learning in K-12 education: A systematic review of SSCI articles during 2009–2022. Journal of Computer Assisted Learning. https://doi.org/10.1111/jcal.12944
Zhai, C., & Wibowo, S. (2022). A systematic review on cross-culture, humor and empathy dimensions in conversational chatbots: The case of second language acquisition. Heliyon, 8(12), 1–13. https://doi.org/10.1016/j.heliyon.2022.e12056
*Zhang, R. F., Zou, D., & Gary, C. (2023). Chatbot-based training on logical fallacy in EFL argumentative writing. Innovation in Language Learning and Teaching. https://doi.org/10.1080/17501229.2023.2197417
Zhao, Y., Wang, N., Li, Y., Zhou, R., & Li, S. (2021). Do cultural differences affect users’ e-learning adoption? A meta-analysis. British Journal of Educational Technology, 52(1), 20–41. https://doi.org/10.1111/bjet.13002
Funding
This study was funded by Scientific Research Funds from Hunan Provincial Ministry of Education (Grant No. 23A0040).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wu, X., Li, R. Unraveling Effects of AI Chatbots on EFL Learners’ Language Skill Development: A Meta-analysis. Asia-Pacific Edu Res (2024). https://doi.org/10.1007/s40299-024-00853-2
Accepted:
Published:
DOI: https://doi.org/10.1007/s40299-024-00853-2