Skip to main content

Advertisement

SpringerLink
Log in

Making FAQ Chatbots More Inclusive: An Examination of Non-Native English Users’ Interactions with New Technology in Massive Open Online Courses

  • Article
  • Published:
International Journal of Artificial Intelligence in Education Aims and scope Submit manuscript

Abstract

In this study, we examined interaction behaviors between a small number of participants in two massive open online courses (MOOCs) and an FAQ chatbot, focusing on the participants’ native language markers. We used a binary native language marker (non-native English user vs. native English user) to distinguish between two groups in this multiple case study, which is based on the knowledge gained from a previous study reporting students’ significantly different experiences based on the two language markers (Han & Lee, Computer & Education, 179, Article 104395, 2022). We utilized a multimodal computer-mediated communication approach as a research guiding tool for examining the data—chatbot logs and the students’ self-reported open-ended responses. We analyzed 42 students’ (non-native English users: n = 27 and native English users: n = 15) precise language uses and constructional practices during student–chatbot interactions to investigate the similarities and differences between the two cases. Most importantly, we focused on the experiences of non-native English users with newly adopted technology, such as chatbots, highlighting a lack of attention to the challenges this group faces despite comprising the majority of MOOC students. By comparing the interactions between the two groups, we identified various student attitudes towards the chatbot with accompanying descriptions, hypothesizing that non-native English users’ positioning effects might have contributed to their negative experiences with the chatbot. We also found that misinterpreted contextualized cues could influence non-native English users more adversely than native English users from an interactional sociolinguistics perspective. Thus, we provide chatbot response design strategies that could support non-native English users better, such as informing users of the chatbot’s capabilities and limitations as early as possible during interactions. We argue that concentrating on the experiences of non-native English users with chatbots promotes a more inclusive learning environment regarding new technologies employed in MOOCs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Aleven, V. A., & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based cognitive tutor. Cognitive Science, 26(2), 147–179. https://doi.org/10.1207/s15516709cog2602_1

    Article  Google Scholar 

  • Barber, J. (2013). The new cyberversity. International Journal of Innovation, Creativity and Change, 1(1), 7–14. https://www.ijicc.net/images/Vol1iss1/The_New_Cyberversity.pdf

  • Bayeck, R. Y., & Choi, J. (2018). The influence of national culture on educational videos: The case of MOOCs. International Review of Research in Open and Distributed Learning, 19(1). https://doi.org/10.19173/irrodl.v19i1.2729

  • Bennett, G. A. (2018). Conversational style: Beyond the nuts and bolts of conversation. In R. J. Moore, M. H. Szymanski, R. Arar, & G. J. Ren (Eds.), Studies in Conversational UX Design (pp. 161–180). Springer International Publishing. https://doi.org/10.1007/978-3-319-95579-7_8

  • Bhattacharya, K. (2017). Fundamentals of qualitative research: A practical guide. Routledge. https://doi.org/10.4324/9781315231747

    Article  Google Scholar 

  • Boonstra, L. (2021). Dialogflow essentials concepts. In The Definitive Guide to Conversational AI with Dialogflow and Google Cloud (pp. 59–91). Apress. https://doi.org/10.1007/978-1-4842-7014-1_3

  • Cho, M. H., & Byun, M. (2017). Nonnative English-speaking students’ lived learning experiences with MOOCs in a regular college classroom. International Review of Research in Open and Distance Learning, 18(5), 173–190. https://doi.org/10.19173/irrodl.v18i5.2892

    Article  Google Scholar 

  • Ciechanowski, L., Przegalinska, A., Magnuski, M., & Gloor, P. (2019). In the shades of the uncanny valley: An experimental study of human–chatbot interaction. Future Generation Computer Systems, 92, 539–548. https://doi.org/10.1016/j.future.2018.01.055

    Article  Google Scholar 

  • Colas, J. F., Sloep, P. B., & Garreta-Domingo, M. (2016). The effect of multilingual facilitation on active participation in MOOCs. International Review of Research in Open and Distributed Learning, 17(4), 280–314. https://doi.org/10.19173/irrodl.v17i4.2470

    Article  Google Scholar 

  • Collopy, F. (1996). Biases in retrospective self-reports of time use: An empirical study of computer users. Management Science, 42(5), 758–767. https://doi.org/10.1287/mnsc.42.5.758

    Article  MATH  Google Scholar 

  • Dale, R. (2016). The return of the chatbots. Natural Language Engineering, 22(5), 811–817. https://doi.org/10.1017/S1351324916000243

    Article  Google Scholar 

  • Davies, B., & Harré, R. (1990). Positioning: The discursive production of selves. Journal for the Theory of Social Behaviour, 20(1), 43–63. https://doi.org/10.1111/j.1468-5914.1990.tb00174.x

    Article  Google Scholar 

  • DeLone, W. H., & McLean, E. R. (1992). Information systems success: The quest for the dependent variable. Information Systems Research, 3(1), 60–95. https://doi.org/10.1287/isre.3.1.60

    Article  Google Scholar 

  • Dialogflow Essentials documentation. (n.d.). Google Cloud. Retrieved July 28, 2021, from https://cloud.google.com/dialogflow/es/docs.

  • Duru, I., Sunar, A. S., White, S., Diri, B., & Dogan, G. (2019). A case study on English as a second language speakers for sustainable MOOC study. Sustainability, 11(10), Article 2808. https://doi.org/10.3390/su11102808

    Article  Google Scholar 

  • D’Silva, G., Jani, M., Jadhav, V., Bhoir, A., & Amin, P. (2020). Career Counselling Chatbot Using Cognitive Science and Artificial Intelligence. In Advanced Computing Technologies and Applications (pp. 1–9). Springer. https://doi.org/10.1007/978-981-15-3242-9_1

  • Engle, D., Mankoff, C., & Carbrey, J. (2015). Coursera’s introductory human physiology course: Factors that characterize successful completion of a MOOC. International Review of Research in Open and Distributed Learning, 16(2). https://doi.org/10.19173/irrodl.v16i2.2010

  • Gauthier, A., Corrin, M., & Jenkinson, J. (2015). Exploring the influence of game design on learning and voluntary use in an online vascular anatomy study aid. Computers & Education, 87, 24–34. https://doi.org/10.1016/j.compedu.2015.03.017

    Article  Google Scholar 

  • Goel, A. K., & Polepeddi, L.. (2019). Jill Watson. In C. Dede, J. Richards, & B. Saxberg (Eds.) Learning Engineering for Online Education (pp. 120–143). Routledge. https://doi.org/10.4324/9781351186193-7

  • Grudin, J., & Jacques, R. (2019). Chatbots, humbots, and the quest for artificial general intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–11). https://doi.org/10.1145/3290605.3300439

  • Gumperz, J. J. (1982). Discourse strategies. Cambridge University Press.

    Book  Google Scholar 

  • Gumperz, J. J. (2005). Interactional sociolinguistics: A personal perspective. In D. Schiffrin, D. Tannen, & H. E. Hamilton (Eds.), The handbook of discourse analysis (pp. 215–228). Blackwell.

    Chapter  Google Scholar 

  • Gupta, S., & Chen, Y. (2022). Supporting inclusive learning using chatbots? A chatbot-led interview study. Journal of Information Systems Education, 33(1), 98–108.

    Google Scholar 

  • Han, S., & Lee, M. K. (2022). FAQ chatbot and inclusive learning in massive open online courses. Computer & Education, 179, Article 104395. https://doi.org/10.1016/j.compedu.2021.104395

    Article  Google Scholar 

  • Harré, R., & Van Langenhove, L. (1999). Positioning theory: Moral contexts of intentional action. Blackwell.

    Google Scholar 

  • Herring, S. C. (2004). Computer-mediated discourse analysis: An approach to researching online behavior. In S. A. Barab, R. Kling, & J. H. Gray (Eds.), Designing for Virtual Communities in the Service of Learning (pp. 338–376). Cambridge University Press.

    Chapter  Google Scholar 

  • Herring S. C. (2019) The coevolution of computer-mediated communication and computer-mediated discourse analysis. In P. Bou-Franch, & P. Garcés-Conejos Blitvich (Eds.), Analyzing Digital Discourse (pp. 25–67). Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-319-92663-6_2

  • Holstein, K., McLaren, B. M., & Aleven, V. (2019). Designing for complementarity: Teacher and student needs for orchestration support in ai-enhanced classrooms. In International Conference on Artificial Intelligence in Education (pp. 157–171). Springer, Cham.

  • Horn, B., Hoover, A. K., Barnes, J., Folajimi, Y., Smith, G., & Harteveld, C. (2016). Opening the Black Box of Play: Strategy Analysis of an Educational Game. Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play, 142–153. https://doi.org/10.1145/2967934.2968109

  • Huang, R. (2016). RQDA: R-based qualitative data analysis. R package version 0.2–8. http://rqda.r-forge.r-project.org/. Accessed 1 May 2020.

  • Huang, W., Hew, K. F., & Gonda, D. E. (2019). Designing and evaluating three chatbot-enhanced activities for a flipped graduate course. International Journal of Mechanical Engineering and Robotics Research, 8(5), 6. https://doi.org/10.18178/ijmerr.8.5.813-818

    Article  Google Scholar 

  • Ji, S., & Yuan, T. (2022). Conversational intelligent tutoring systems for online learning: What do students and tutors say? IEEE Global Engineering Education Conference (EDUCON), 2022, 292–298. https://doi.org/10.1109/EDUCON52537.2022.9766567

    Article  Google Scholar 

  • Joksimovic, S., Gasevic, D., & Hatala, M. (2014). Learning analytics for networked learning models. Journal of Learning Analytics, 1(3), 191-194. https://doi.org/10.18608/jla.2014.13.20

    Article  Google Scholar 

  • Jordan, K. (2014). Initial trends in enrolment and completion of massive open online courses. International Review of Research in Open and Distance Learning, 15(1), 133–160. https://doi.org/10.19173/irrodl.v15i1.1651

    Article  Google Scholar 

  • Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., & Siemens, G. (2016). Towards automated content analysis of discussion transcripts: A cognitive presence case. In Proceedings of the sixth international conference on learning analytics & knowledge (pp. 15–24). https://doi.org/10.1145/2883851.2883950

  • Lee, S., Lee, N., & Sah, Y. J. (2020). Perceiving a mind in a chatbot: Effect of mind perception and social cues on co-presence, closeness, and intention to use. International Journal of Human-Computer Interaction, 36(10), 930–940. https://doi.org/10.1080/10447318.2019.1699748

    Article  Google Scholar 

  • Li, C. H., Yeh, S. F., Chang, T. J., Tsai, M. H., Chen, K., & Chang, Y. J. (2020, April). A conversation analysis of non-progress and coping strategies with a banking task-oriented chatbot. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–12). https://doi.org/10.1145/3313831.3376209

  • Liao, Q. V., Hussain, M. M., Chandar, P., Davis, M., Khazaeni, Y., Crasso, M. P., ..., & Geyer, W. (2018, April). All work and no play?. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1–13). https://doi.org/10.1145/3173574.3173577

  • Luger, E., & Sellen, A. (2016). “Like having a really bad PA”: The gulf between user expectation and experience of conversational agents. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 5286–5297). https://doi.org/10.1145/2858036.2858288

  • Luo, X., Tong, S., Fang, Z., & Qu, Z. (2019). Frontiers: Machines vs. humans: The impact of artificial intelligence chatbot disclosure on customer purchases. Marketing Science, 38(6), 937–947. https://doi.org/10.1287/mksc.2019.1192

    Article  Google Scholar 

  • Mori, M. (1970). Bukimi no tani [the uncanny valley]. Energy, 7, 33–35.

    Google Scholar 

  • Morkes, J., Kernal, H., & Nass, C. (1999). Effects of humor in task-oriented human-computer interaction and computer-mediated communication: A direct test of SRCT theory. Human-Computer Interaction, 14(4), 395–435. https://doi.org/10.1207/S15327051HCI1404_2

    Article  Google Scholar 

  • Murtarelli, G., Gregory, A., & Romenti, S. (2021). A conversation-based perspective for shaping ethical human–machine interactions: The particular challenge of chatbots. Journal of Business Research, 129, 927–935. https://doi.org/10.1016/j.jbusres.2020.09.018

    Article  Google Scholar 

  • Nass, C., & Moon, Y. (2000). Machines and mindlessness: Social responses to computers. Journal of Social Issues, 56(1), 81–103. https://doi.org/10.1111/0022-4537.00153

    Article  Google Scholar 

  • Nass, C., Moon, Y., & Carney, P. (1999). Are respondents polite to computers? Social desirability and direct responses to computers. Journal of Applied Social Psychology, 29(5), 1093–1110.

    Article  Google Scholar 

  • Nass, C., Steuer, J. S., Henriksen, L., & Dryer, D. C. (1994). Machines and social attributions: Performance assessments of computers subsequent to “self-” or “other-” evaluations. International Journal of Human-Computer Studies, 40, 543–559.

    Article  Google Scholar 

  • Psathas, G. (1995). Conversation analysis. SAGE Publications, Inc. https://doi.org/10.4135/9781412983792

  • Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people and places. Cambridge University Press.

    Google Scholar 

  • Reilly, E. D., Williams, K. M., Stafford, R. E., Corliss, S. B., Walkow, J. C., & Kidwell, D. K. (2016). Global times call for global measures: Investigating automated essay scoring in linguistically diverse MOOCs. Online Learning, 20(2), 97–109. https://doi.org/10.24059/olj.v20i2.638

    Article  Google Scholar 

  • Richards, N. M., & King, J. H. (2014). Big data ethics. Wake Forest Law Review, 49(2), 393–432.

    Google Scholar 

  • Richterich, A. (2018). The big data agenda: Data ethics and critical data studies. University of Westminster Press. https://doi.org/10.16997/book14

  • Rubin, V. L., Chen, Y., & Thorimbert, L. M. (2010). Artificially intelligent conversational agents in libraries. Library Hi Tech, 28(4), 496–522. https://doi.org/10.1108/07378831011096196

    Article  Google Scholar 

  • Sanchez-Gordon, S., & Luján-Mora, S. (2015). Accessible blended learning for non-native speakers using MOOCs. In 2015 International Conference on Interactive Collaborative and Blended Learning (ICBL) (pp. 19–24). IEEE.

  • Sandoval, Z. V. (2018). Design and implementation of a chatbot in online higher education settings. Issues in Information Systems, 19(4). https://doi.org/10.48009/4_iis_2018_44-52

  • Schuetzler, R. M., Grimes, G. M., & Scott Giboney, J. (2020). The impact of chatbot conversational skill on engagement and perceived humanness. Journal of Management Information Systems, 37(3), 875–900. https://doi.org/10.1080/07421222.2020.1790204

    Article  Google Scholar 

  • Shawar, B. A., & Atwell, E. (2007). Chatbots: Are they really useful?. In Ldv forum (pp. 29–49).

  • Shumanov, M., & Johnson, L. (2021). Making conversations with chatbots more personalized. Computers in Human Behavior, 117, Article 106627. https://doi.org/10.1016/j.chb.2020.106627

    Article  Google Scholar 

  • Song, D., Rice, M., & Oh, E. Y. (2019). Participation in online courses and interaction with a virtual agent. International Review of Research in Open and Distributed Learning, 20(1). https://doi.org/10.19173/irrodl.v20i1.3998

  • Straub, D., Limayem, M., & Karahanna-Evaristo, E. (1995). Measuring system usage: Implications for IS theory testing. Management Science, 41(8), 1328–1342. https://doi.org/10.1287/mnsc.41.8.1328

    Article  Google Scholar 

  • Tahirsylaj, A., Mann, B., & Matson, J. (2018). Teaching creativity at scale: Overcoming language barriers in a MOOC. International Journal of Innovation, Creativity and Change, 4(2), 1–19.

    Google Scholar 

  • Tannen, D. (ed.). (1993). Framing in discourse. Oxford University Press on Demand.

  • Tannen, D. (2005). Conversational style: Analyzing talk among friends. Oxford University Press.

    Google Scholar 

  • Trice, A. W., & Treacy, M. E. (1988). Utilization as a dependent variable in MIS research. ACM SIGMIS Database: The Database for Advances in Information Systems, 19(3–4), 33–41. https://doi.org/10.1145/65766.65771

    Article  Google Scholar 

  • Vilaza, G. N., & McCashin, D. (2021). Is the automation of digital mental health ethical? Applying an ethical framework to chatbots for cognitive behavior therapy. Frontiers in Digital Health, 3. https://doi.org/10.3389/fdgth.2021.689736

  • Verschuren, P. (2003). Case study as a research strategy: Some ambiguities and opportunities. International Journal of Social Research Methodology, 6(2), 21–139.

    Article  Google Scholar 

  • Wambsganss, T., Winkler, R., Söllner, M., & Leimeister, J. M. (2020). A conversational agent to improve response quality in course evaluations. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–9). https://doi.org/10.1145/3334480.3382805

  • Wang, J., Hwang, G. H., & Chang, C. Y. (2021a). Directions of the 100 most cited chatbot-related human behavior research: A review of academic publications. Computers and Education: Artificial Intelligence, 2, Article 100023. https://doi.org/10.1016/j.caeai.2021.100023

    Article  Google Scholar 

  • Wang, Q., Jing, S., Camacho, I., Joyner, D., & Goel, A. (2020). Jill Watson SA: Design and evaluation of a virtual agent to build communities among online learners. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–8). https://doi.org/10.1145/3334480.3382878

  • Wang, Q., Saha, K., Gregori, E., Joyner, D., & Goel, A. (2021b). Towards mutual theory of mind in human-AI interaction: How language reflects what students perceive about a virtual teaching assistant. In Proceedings of the 2021b CHI Conference on Human Factors in Computing Systems (pp. 1–14). https://doi.org/10.1145/3411764.3445645

  • Wang, Y. F., & Petrina, S. (2013). Using learning analytics to understand the design of an intelligent language tutor–Chatbot lucy. International Journal of Advanced Computer Science and Applications (IJACSA), 4(11), Article 11. https://doi.org/10.14569/IJACSA.2013.041117

    Article  Google Scholar 

  • Winkler, R., Hobert, S., Salovaara, A., Söllner, M., & Leimeister, J. M. (2020). Sara, the lecturer: Improving learning in online education with a scaffolding-based conversational agent. In Proceedings of the 2020 CHI conference on human factors in computing systems (pp. 1–14). https://doi.org/10.1145/3313831.3376781

  • Yang, H. D., & Yoo, Y. (2004). It’s all about attitude: Revisiting the technology acceptance model. Decision Support Systems, 38(1), 19–31. https://doi.org/10.1016/S0167-9236(03)00062-9

    Article  Google Scholar 

  • Yin, R. K. (2017). Case study research and applications: Design and methods. SAGE Publications.

    Google Scholar 

  • Yoo, J., & Kim, J. (2014). Can online discussion participation predict group project performance? Investigating the roles of linguistic features and participation patterns. International Journal of Artificial Intelligence in Education, 24(1), 8–32. https://doi.org/10.1007/s40593-013-0010-8

    Article  Google Scholar 

  • Zamora, J. (2017). I’m sorry, dave, i’m afraid i can’t do that: Chatbot perception and expectations. In Proceedings of the 5th international conference on human agent interaction (pp. 253–260). https://doi.org/10.1145/3125739.3125766

  • Zhu, M., Herring, S. C., & Bonk, C. J. (2019). Exploring presence in online learning through three forms of computer-mediated discourse analysis. Distance Education, 40(2), 205–225. https://doi.org/10.1080/01587919.2019.1600365

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Curriculum and Instruction, The University of Texas at Austin, Austin, TX, USA

    Songhee Han, Min Liu, Zilong Pan, Ying Cai & Peixia Shao

Authors
  1. Songhee Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Min Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zilong Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ying Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Peixia Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Songhee Han.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest in this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, S., Liu, M., Pan, Z. et al. Making FAQ Chatbots More Inclusive: An Examination of Non-Native English Users’ Interactions with New Technology in Massive Open Online Courses. Int J Artif Intell Educ 33, 752–780 (2023). https://doi.org/10.1007/s40593-022-00311-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40593-022-00311-4

Keywords

Search

Navigation