What matters in AI-supported learning: A study of human-AI interactions in language learning using cluster analysis and epistemic network analysis

https://doi.org/10.1016/j.compedu.2022.104703Get rights and content

Highlights

  • Community of inquiry and students' approaches to learning are used to analyze human-AI interactions.

  • Four clusters of students are identified with distinct ways of interacting with AI.

  • Not everyone can benefit from the potential promised by AI.

  • The deep learning approach may enhance the human-AI learning community, leading to high performance.

  • Passively or mechanically following AI's instruction may weaken the human-AI learning community.

Abstract

This study investigates how students interact with artificial intelligence (AI) for English as a Foreign Language (EFL) learning and what matters in AI-supported EFL learning. It was conducted in naturalistic learning settings, involving sixteen primary school students and lasting approximately three months. The students' usage data of an AI agent and their reflection essays about the interactions with the AI agent were analyzed using cluster analysis and epistemic network analysis based on the frameworks of community of inquiry and students' approaches to learning. The results suggest four clusters of students, each with its distinct way of interacting with AI for language learning. More importantly, the comparisons of the four clusters of students reveal that even in AI-supported learning, not everyone can benefit from the potential promised by AI. The deep approach to AI-supported learning may amplify the benefits of AI's personalized guidance and strengthen the sense of the human-AI learning community. Passively or mechanically following AI's instruction, albeit with high levels of participation, may decrease the sense of the human-AI learning community and eventually lead to low performance. This study contributes to and has implications for the educational implementation of AI, as well as the facilitation and graphical representation of learner-AI interactions in educational settings.

Introduction

Studies exploring the use of affordable AI agents such as chatbots and conversational agents for second language (L2) learning have been increasing in recent years (Jeon, 2022; Lin & Mubarok, 2021; Underwood, 2017). AI agents provide students with ample opportunities to practice target languages in relatively stress-free learning environments, and particularly, offer personalized instruction for individual students in large classes (Dizon, 2020; Moussalli & Cardoso, 2020). Students have mostly reported that they enjoy and feel more relaxed about speaking to AI in L2 than to real humans (Wang et al., 2022; Tai & Chen, 2020). These benefits associated with AI agents offer promising solutions to intricate problems encountered in conventional L2 classrooms, such as inadequate speaking practice opportunities and limited class time (e.g., de Vries et al., 2015), unwillingness to speak (e.g., Tai & Chen, 2020), and lack of personalized feedback to every student (e.g., Luo, 2016).

Despite the benefits associated with AI-supported language learning, there is a noteworthy dearth of knowledge regarding how students interact with AI agents for language learning and what differences may exist among distinct types of students in human-AI interactions. Without such knowledge, the mechanism of AI-supported language learning remains a black box, that is, while students’ initial language competence at one end and their final competence at the other have been examined, what happens in between is largely unknown. Consequently, few insights can be gained to improve the design and implementation of AI agents in language learning as well as the pedagogy of AI-supported language learning.

To address this gap, this study investigated the human-AI interactions in an English as a Foreign Language (EFL) class involving 16 sixth-grade students in a primary school. An AI agent called the “AI coach” was used in this study. The AI coach was a virtual AI agent installed on mobile devices and was embodied as a female teacher. It was specifically created for EFL learning, supporting unlimited English practice and providing personalized feedback on students’ speaking.

Considering that prior studies in AI-supported language learning have anthropomorphized AI applications as peers, teachers, tutors, and teaching assistants for learners (Engwall & Lopes, 2020; Randall, 2019), students and AI may form a learning community where the social, cognitive, and teaching presences students perceive in interacting with AI can forge meaningful learning experiences and facilitate L2 acquisition (Jeon, 2022; Yu & Li, 2022). As such, theoretically, community of inquiry (CoI; Garrison et al., 2010; Garrison & Arbaugh, 2007) serves as a feasible framework for analyzing human-AI interactions to reveal how learning happens in AI-supported language learning (Wang et al., 2022; Yu & Li, 2022). Nonetheless, young learners may not necessarily be competent to use digital technologies, including AI, effectively for learning, though they are often perceived to be tech-savvy (Qi, 2019; Thompson, 2013). Substantial research (e.g., Niu et al., 2022; Ellis & Bliuc, 2019; Lindblom-Ylänne et al., 2019) has shown that the ways in which students approach online or offline learning play a crucial role in determining their learning experiences and performance. Students' approaches to learning (SAL) framework has been found to effectively explain why some students are more successful than others in school (Ellis & Bliuc, 2019). Though SAL have been widely applied in a variety of learning contexts (e.g., Biggs & Tang, 2011; Ellis & Bliuc, 2019; Hailikari & Parpala, 2014), it is still unknown how students’ approaches to AI-supported learning function in a human-AI community. Therefore, building on the combined frameworks of CoI and SAL, this study employed cluster analysis and epistemic network analysis (ENA) to provide insights into how distinct types of students interacted with the AI coach for EFL learning. Overall, this study sought to address the following two research questions:

  • (RQ1) How many distinct types of learners can be identified with respect to the AI usage?

  • (RQ2) How do distinct types of learners interact with AI for language learning and what are their differences in human-AI interactions, if any?

Section snippets

AI agents for language learning

AI agents in this study refer to entities that are capable of performing tasks related to intelligent beings through reasoning, learning, and expressing themselves, to a certain extent (Wang et al., 2022; Berendt et al., 2020). They are created in different forms (Randall, 2019), for instance, anthropomorphic (with human-like appearances), cartoon-like (with exaggerated cartoon features), and mechanomorphic (with machine-like features). Based on the visibility of AI agents, they can also be

Participants and research context

The present study was conducted in a 6th grade EFL class in an urban primary school in China, lasting approximately three months. The research context in this study was similar to Wang et al., 2022. The students started to learn English in Year 1 of primary school. Nonetheless, their English proficiency was normally at very basic levels due to limited exposure to authentic L2 environments and exam-oriented education. The AI coach (kouyu100.com) was purchased by the school to cope with the lack

Results

In this section, the results related to the cluster analysis were first reported, followed by the ENA results of different clusters of students’ retrospective interactions with the AI coach.

Discussion

Following the CoI and SAL frameworks, this study combined cluster analysis and epistemic network analysis to investigate how students of distinct types interacted with the AI coach for L2 learning and to reveal what matters in AI-supported language learning. Overall, three key findings can be derived.

Contributions and implications

This study contributes to AI-supported language learning in several ways. First, the present study advances the state-of-the-art understanding of how learners interact with AI technologies for language learning. Though AI has been considered useful and effective in improving learners' L2 competence and forging positive learning experiences, little is known regarding how human-AI interactions actually function for L2 learning. Though few studies (e.g., Lee & Jeon, 2022; Moussalli & Cardoso, 2020

Limitations and future research

The findings of this study should be interpreted with the following limitations in mind. First, this study involved a small number of participants studying EFL and was conducted in a primary school. Future research seeking to generalize the research findings can be conducted with more participants from diverse disciplines and school levels. Second, the present study mainly relied on two data sources: students' actual usage data and their reflection essays on their using experiences. To obtain

Credit author statement

Xinghua Wang: Conceptualization, Investigation, Methodology, Software, Formal analysis, Writing-original draft; Qian Liu: Conceptualization, Investigation, Formal analysis, Writing-original draft; Hui Pang: Resources, Data curation, Methodology, Writing-original draft, Writing-review & editing; Seng Chee Tan: Conceptualization, Methodology, Writing-review & editing. Jun Lei: Conceptualization, Methodology, Writing-review & editing. Matthew P. Wallace: Conceptualization, Methodology,

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

References (70)

  • G. Awada

    Robotics and Weblog integration into EFL curriculum of middle and secondary schools: Exploratory study

    Language Teaching Research

    (2022)
  • M. Benassi et al.

    Using two-step cluster analysis and latent class cluster analysis to classify the cognitive heterogeneity of cross-diagnostic psychiatric inpatients

    Frontiers in Psychology

    (2020)
  • B. Berendt et al.

    AI in education: Learner choice and fundamental rights

    Learning, Media and Technology

    (2020)
  • R. van den Berghe et al.

    Social robots for language learning: A review

    Review of Educational Research

    (2019)
  • J. Biggs

    What do inventories of students' learning processes really measure? A theoretical review and clarification

    British Journal of Educational Psychology

    (1993)
  • J. Biggs et al.

    Teaching for quality learning at university

    (2011)
  • S. Cai et al.

    Tablet‐based AR technology: Impacts on students' conceptions and approaches to learning mathematics according to their self‐efficacy

    British Journal of Educational Technology

    (2019)
  • K.H. Cheng

    Exploring parents' conceptions of augmented reality learning and approaches to learning by augmented reality with their children

    Journal of Educational Computing Research

    (2017)
  • H.H.-J. Chen et al.

    Investigating college EFL learners' perceptions toward the use of Google Assistant for foreign language learning

    Interactive Learning Environments

    (2020)
  • G. Dizon

    Evaluating intelligent personal assistants for L2 listening and speaking development

    Language, Learning and Technology

    (2020)
  • R.A. Ellis et al.

    Exploring new elements of the student approaches to learning framework: The role of online learning technologies in student learning

    Active Learning in Higher Education

    (2019)
  • O. Engwall et al.

    Interaction and collaboration in robot-assisted language learning for adults

    Computer Assisted Language Learning

    (2020)
  • J. Fan et al.

    Interpretable MOOC recommendation: A multi-attention network for personalized learning behavior analysis

    Internet Research

    (2021)
  • C.V. Felix

    The role of the teacher and AI in education

  • D.R. Garrison et al.

    Facilitating cognitive presence in online learning: Interaction is not enough

    American Journal of Distance Education

    (2005)
  • G. Gibbs et al.

    The impact of training of university teachers on their teaching skills, their approach to teaching and the approach to learning of their students

    Active Learning in Higher Education

    (2004)
  • T.K. Hailikari et al.

    What impedes or enhances my studying? The interrelation between approaches to learning, factors influencing study progress and earned credits

    Teaching in Higher Education

    (2014)
  • Y. Hamada

    Shadowing: Who benefits and how? Uncovering a booming EFL teaching technique for listening comprehension

    Language Teaching Research

    (2016)
  • T. Heift

    Learner control and error correction in ICALL: Browsers, peekers, and adamants

    Calico Journal

    (2002)
  • T. Heift et al.

    Errors and intelligence in computer-assisted language learning: Parsers and pedagogues

    (2007)
  • K.J. Herrmann et al.

    Investigating the relationships between approaches to learning, learner identities and academic achievement in higher education

    Higher Education

    (2017)
  • W.-Y. Hwang et al.

    Learning behavior analysis of a ubiquitous situated reflective learning system with application to life science and technology teaching

    Educational Technology & Society

    (2018)
  • J. Jeon

    Exploring AI chatbot affordances in the EFL classroom: Young learners' experiences and perspectives

    Computer Assisted Language Learning

    (2022)
  • L. Kaufman et al.

    Finding groups in data: An introduction to cluster analysis

    (2009)
  • P. Kent et al.

    A comparison of three clustering methods for finding subgroups in MRI, SMS or clinical data: SPSS TwoStep cluster analysis, latent gold and SNOB

    BMC Medical Research Methodology

    (2014)
  • Cited by (31)

    View all citing articles on Scopus
    1

    Xinghua Wang and Qian Liu contribute equally to this study and share first authorship.

    View full text