Skip to main content
SpringerLink
Log in

CBET: design and evaluation of a domain-specific chatbot for mobile learning

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

The popularity of mobile devices and conversational agents in recent years has seen wide use of chatbots in different educational scenarios. In relation to the advances in mobile devices and conversational agents, there are few research works concerning the design and evaluation of domain-specific chatbots to fulfill the demand of mobile learning. To address this issue, we propose an agent-based conceptual architecture to develop a domain-specific chatbot for mobile learning. We extend the open-domain DeepQA agent to make it sensitive to restricted domain questions by building a domain-specific gate, and employ WeChat as user interface. To evaluate our chatbot, subjective and objective criteria are employed to assess its effectiveness. Additionally, its usability evaluation proceeds with system usability scale questionnaire and net promoter score simultaneously. In total, 18 domain experts participated in the evaluation of effectiveness, and 52 participants were involved in the evaluation of usability. Based on the evaluation results, we conclude that our chatbot can serve as an effective information retrieval tool in a specific domain. The perceived usability of our chatbot tends to be moderate and marginal and has positively affected the promotion of our chatbot for mobile learning. This paper contributes to the educative application of chatbots in specific subject fields.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. This domain-specific corpus is available at https://github.com/JimSow/CBET-Corpus.

  2. This dataset is available at https://github.com/JimSow/CBET-Corpus.

  3. The official Web site of ACO is available at http://aco.ccnu.edu.cn.

References

  1. Chakrabarti, C., Luger, G.F.: Artificial conversations for customer service chatter bots: architecture, algorithms, and evaluation metrics. Expert Syst. Appl. 42(20), 6878–6897 (2015). https://doi.org/10.1016/j.eswa.2015.04.067

    Article  Google Scholar 

  2. Crutzen, R., Peters, G.J.Y., Portugal, S.D., Fisser, E.M., Grolleman, J.J.: An artificially intelligent chat agent that answers adolescents’ questions related to sex, drugs, and alcohol: an exploratory study. J. Adolesc. Health 48(5), 514–519 (2011). https://doi.org/10.1016/j.jadohealth.2010.09.002

    Article  Google Scholar 

  3. Griol, D., Callejas, Z.: Mobile conversational agents for context-aware care applications. Cogn. Comput. 8(2), 336–356 (2016). https://doi.org/10.1007/s12559-015-9352-x

    Article  Google Scholar 

  4. Griol, D., Callejas, Z.: An architecture to develop multimodal educative applications with chatbots. Int. J. Adv. Rob. Syst. 10(3), 175–190 (2013). https://doi.org/10.5772/55791

    Article  Google Scholar 

  5. Kerly, A., Hall, P., Bull, S.: Bringing chatbots into education: towards natural language negotiation of open learner models. Knowl. Based Syst. 20(2), 177–185 (2007). https://doi.org/10.1016/j.knosys.2006.11.014

    Article  Google Scholar 

  6. Kerly, A., Ellis, R., Bull, S.: CALMsystem: a conversational agent for learner modelling. In: International Conference on Innovative Techniques and Applications of Artificial Intelligence, pp. 89–102. Springer (2008). https://doi.org/10.1007/978-1-84800-086-5_7

  7. Benotti, L., Martínez, M.C., Schapachnik, F.: Engaging high school students using chatbots. In: Proceedings of the 2014 Conference on Innovation and Technology in Computer Science Education, pp. 63–68. ACM, New York, NY, USA (2014)

  8. Jia, J.: CSIEC: a computer assisted English learning chatbot based on textual knowledge and reasoning. Knowl. Based Syst. 22(4), 249–255 (2009). https://doi.org/10.1016/j.knosys.2008.09.001

    Article  Google Scholar 

  9. Goda, Y., Yamada, M., Matsukawa, H., Hata, K., Yasunami, S.: Conversation with a chatbot before an online EFL group discussion and the effects on critical thinking. J. Inf. Syst. Educ. 13(1), 1–7 (2014). https://doi.org/10.12937/ejsise.13.1

    Article  Google Scholar 

  10. Coniam, D.: The linguistic accuracy of chatbots: usability from an ESL perspective. Text Talk. 34(5), 545–567 (2014). https://doi.org/10.1515/text-2014-0018

    Article  Google Scholar 

  11. Pérez-Marín, D., Boza, A.: A procedure to create a pedagogic conversational agent in secondary physics and chemistry education. Int. J. Inf. Commun. Technol. Educ. 9(4), 94–112 (2013). https://doi.org/10.4018/ijicte.2013100107

    Article  Google Scholar 

  12. Graesser, A.C., Li, H., Forsyth, C.: Learning by communicating in natural language with conversational agents. Curr. Dir. Psychol. Sci. 23(5), 374–380 (2014). https://doi.org/10.1177/0963721414540680

    Article  Google Scholar 

  13. Kumar, R., Rose, C.P.: Architecture for building conversational agents that support collaborative learning. IEEE Trans. Learn. Technol. 4(1), 21–34 (2011). https://doi.org/10.1109/TLT.2010.41

    Article  Google Scholar 

  14. Dyke, G., Adamson, D., Howley, I., Rosé, C.P.: Enhancing scientific reasoning and discussion with conversational agents. IEEE Trans. Learn. Technol. 6(3), 240–247 (2013). https://doi.org/10.1109/TLT.2013.25

    Article  Google Scholar 

  15. Tegos, S., Demetriadis, S., Karakostas, A.: Promoting academically productive talk with conversational agent interventions in collaborative learning settings. Comput. Educ. 87, 309–325 (2015). https://doi.org/10.1016/j.compedu.2015.07.014

    Article  Google Scholar 

  16. Tegos, S., Demetriadis, S., Papadopoulos, P.M., Weinberger, A.: Conversational agents for academically productive talk: a comparison of directed and undirected agent interventions. Int. J. Comput. Support. Collab. Learn. 11(4), 417–440 (2016). https://doi.org/10.1007/s11412-016-9246-2

    Article  Google Scholar 

  17. Preece, A., Webberley, W., Braines, D., Zaroukian, E.G., Bakdash, J.Z.: SHERLOCK: experimental evaluation of a conversational agent for mobile information tasks. IEEE Trans. Hum. Mach. Syst. 47(6), 1017–1028 (2017). https://doi.org/10.1109/THMS.2017.2700625

    Article  Google Scholar 

  18. Wu, W.H., Wu, Y.C.J., Chen, C.Y., Kao, H.Y., Lin, C.H., Huang, S.H.: Review of trends from mobile learning studies: a meta-analysis. Comput. Educ. 59(2), 817–827 (2012). https://doi.org/10.1016/j.compedu.2012.03.016

    Article  Google Scholar 

  19. Becker, S.A., Cummins, M., Davis, A., Freeman, A., Glesinger Hall, C., Ananthanarayanan, V.: The NMC Horizon Report: 2017 Higher Education Edition. The New Media Consortium, Austin (2017)

    Google Scholar 

  20. Liu, Q., Wu, L., Yang, Z., Liu, Y.: Domain phrase identification using atomic word formation in Chinese text. Knowl. Based Syst. 24(8), 1254–1260 (2011). https://doi.org/10.1016/j.knosys.2011.06.001

    Article  Google Scholar 

  21. Wu, L., Liu, Q., Zhao, G., Huang, H., Huang, T.: Thesaurus dataset of educational technology in Chinese. Br. J. Edu. Technol. 46(5), 1118–1122 (2015). https://doi.org/10.1111/bjet.12298

    Article  Google Scholar 

  22. Hwang, G.J., Tsai, C.C.: Research trends in mobile and ubiquitous learning: a review of publications in selected journals from 2001 to 2010. Br. J. Educ. Technol. 42(4), E65–E70 (2011). https://doi.org/10.1111/j.1467-8535.2011.01183.x

    Article  Google Scholar 

  23. Hung, J.L., Zhang, K.: Examining mobile learning trends 2003–2008: a categorical meta-trend analysis using text mining techniques. J. Comput. High. Educ. 24(1), 1–17 (2012). https://doi.org/10.1007/s12528-011-9044-9

    Article  Google Scholar 

  24. Su, C.H., Cheng, C.H.: A mobile gamification learning system for improving the learning motivation and achievements. J. Comput. Assist. Learn. 31(3), 268–286 (2015). https://doi.org/10.1111/jcal.12088

    Article  Google Scholar 

  25. Yousafzai, A., Chang, V., Gani, A., Noor, R.M.: Multimedia augmented m-learning: issues, trends and open challenges. Int. J. Inf. Manage. 36(5), 784–792 (2016). https://doi.org/10.1016/j.ijinfomgt.2016.05.010

    Article  Google Scholar 

  26. Lowe, R., Pow, N., Serban, I., Pineau, J.: The Ubuntu dialogue corpus: a large dataset for research in unstructured multi-turn dialogue systems. In: Proceedings of the SIGDIAL 2015 Conference, pp. 285–294. Association for Computational Linguistics (2015)

  27. Moreno, R., Mayer, R.: Interactive multimodal learning environments. Educ. Psychol. Rev. 19(3), 309–326 (2007). https://doi.org/10.1007/s10648-007-9047-2

    Article  Google Scholar 

  28. Kim, N.Y.: A study on different types of speech acts in voice-chat between EFL students and a chatbot. Stud. Engl. Educ. 22(3), 81–109 (2017)

    Article  Google Scholar 

  29. Ayedoun, E., Hayashi, Y., Seta, K.: Web-services based conversational agent to encourage willingness to communicate in the EFL context. J. Inf. Syst. Educ. 15(1), 15–27 (2016). https://doi.org/10.12937/ejsise.15.15

    Article  Google Scholar 

  30. Fadhil, A., Villafiorita, A.: An adaptive learning with gamification & conversational UIs: The rise of CiboPoliBot. In: Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization, pp. 408–412. ACM, New York, NY, USA (2017)

  31. Tegos, S., Demetriadis, S., Tsiatsos, T.: An investigation of conversational agent interventions supporting historical reasoning in primary education. In: International Conference on Intelligent Tutoring Systems, pp. 260–266. Springer (2016)

  32. Ardito, C., Costabile, M.F., Marsico, M.D., Lanzilotti, R., Levialdi, S., Roselli, T., et al.: An approach to usability evaluation of e-learning applications. Univ. Access Inf. Soc. 4(3), 270–283 (2006). https://doi.org/10.1007/s10209-005-0008-6

    Article  Google Scholar 

  33. Song, D., Oh, E.Y., Rice, M.: Interacting with a conversational agent system for educational purposes in online courses. In: Proceedings of the 10th International Conference on Human System Interactions, pp. 78–82. IEEE (2017)

  34. Shawar, B. A., Atwell, E.: Different measurements metrics to evaluate a chatbot system. In: Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies, pp. 89–96. Association for Computational Linguistics (2007)

  35. Yu, Z., Xu, Z., Black, A.W., Rudnicky, A.: Chatbot evaluation and database expansion via crowdsourcing. In: Proceedings of the chatbot workshop of LREC, pp. 102–107. Association for Computational Linguistics (2016)

  36. Radziwill, N.M., Benton, M.C.: Evaluating quality of chatbots and intelligent conversational agents. CoRR abs/1704.04579 (2017)

  37. Liu, C.W., Lowe, R., Serban, I.V., Noseworthy, M., Charlin, L., Pineau, J.: How not to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. CoRR abs/1603.08023 (2016)

  38. Orfanou, K., Tselios, N., Katsanos, C.: Perceived usability evaluation of learning management systems: empirical evaluation of the System Usability Scale. Int. Rev. Res. Open Distrib. Learn. 16(2), 227–246 (2015). https://doi.org/10.19173/irrodl.v16i2.1955

    Article  Google Scholar 

  39. Revythi, A., Tselios, N.: Extension of Technology Acceptance Model by using System Usability Scale to assess behavioral intention to use e-learning. CoRR abs/1704.06127

  40. Apdplat.: deep-qa. Retrieved from https://github.com/ysc/QuestionAnsweringSystem/deep-qa (10 October, 2018)

  41. Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A.A., Schlaefer, N.: Building Watson: an overview of the DeepQA project. AI Mag. 31(3), 59–79 (2010). https://doi.org/10.1609/aimag.v31i3.2303

    Article  Google Scholar 

  42. Venkatesh, A., Khatri, C., Ram, A., Guo, F., Gabriel, R., Nagar, A., Goel, R.: On Evaluating and Comparing Conversational Agents. CoRR abs/1801.03625 (2018)

  43. Juszczak, P.: Learning to recognize: A study on one-class classification and active learning. Dissertation, Delft University of Technology (2006)

  44. Zhang, C.: Automatic keyword extraction from documents using conditional random fields. J. Comput. Inf. Syst. 4(3), 1169–1180 (2008)

    Google Scholar 

  45. Désir, C., Bernard, S., Petitjean, C., Heutte, L.: One class random forests. Pattern Recognit. 46(12), 3490–3506 (2013). https://doi.org/10.1016/j.patcog.2013.05.022

    Article  Google Scholar 

  46. Gaudio, R.D., Batista, G., Branco, A.: Coping with highly imbalanced datasets: a case study with definition extraction in a multilingual setting. Nat. Lang. Eng. 20(3), 327–359 (2014). https://doi.org/10.1017/S1351324912000381

    Article  Google Scholar 

  47. Reiplinger, M., Schäfer, U., Wolska, M.: Extracting glossary sentences from scholarly articles: A comparative evaluation of pattern bootstrapping and deep analysis. In: Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, pp. 55–65. Association for Computational Linguistics (2012)

  48. Jin, Y., Kan, M.Y., Ng, J.P., He, X.: Mining scientific terms and their definitions: A study of the ACL anthology. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 780–790. Association for Computational Linguistics (2013)

  49. Abacha, A.B., Zweigenbaum, P.: Automatic extraction of semantic relations between medical entities: a rule based approach. J. Biomed. Semant. 2(5), S4 (2011). https://doi.org/10.1186/2041-1480-2-S5-S4

    Article  Google Scholar 

  50. Punuru, J., Chen, J.: Learning non-taxonomical semantic relations from domain texts. J. Intell. Inf. Syst. 38(1), 191–207 (2012). https://doi.org/10.1007/s10844-011-0149-4

    Article  Google Scholar 

  51. Shen, M., Liu, D.R., Huang, Y.S.: Extracting semantic relations to enrich domain ontologies. J. Intell. Inf. Syst. 39(3), 749–761 (2012). https://doi.org/10.1007/s10844-012-0210-y

    Article  Google Scholar 

  52. Xu, J., Kang, Q., Song, Z., Clarke, C.P.: Applications of mobile social media: WeChat among academic libraries in China. J. Acad. Librariansh. 41(1), 21–30 (2015). https://doi.org/10.1016/j.acalib.2014.10.012

    Article  Google Scholar 

  53. So, S.: Mobile instant messaging support for teaching and learning in higher education. Internet High. Educ. 31, 32–42 (2016). https://doi.org/10.1016/j.iheduc.2016.06.001

    Article  Google Scholar 

  54. Sun, Z., Liu, R., Luo, L., Wu, M., Shi, C.: Exploring collaborative learning effect in blended learning environments. J. Comput. Assist. Learn. 33(6), 575–587 (2017). https://doi.org/10.1111/jcal.12201

    Article  Google Scholar 

  55. Goh, O.S., Ardil, C., Wong, W., Fung, C.C.: A black-box approach for response quality evaluation of conversational agent systems. Int. J. Comput. Intell. 3(3), 195–203 (2007)

    Google Scholar 

  56. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311–318. Association for Computational Linguistics (2002)

  57. Murdock, J.W., Fan, J., Lally, A., Shima, H.: Textual evidence gathering and analysis. IBM J. Res. Dev. 56(3.4), 8:1–8:14 (2012). https://doi.org/10.1147/JRD.2012.2187249

    Article  Google Scholar 

  58. Harrati, N., Bouchrika, I., Tari, A., Ladjailia, A.: Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Comput. Hum. Behav. 61, 463–471 (2016). https://doi.org/10.1016/j.chb.2016.03.051

    Article  Google Scholar 

  59. Kortum, P.T.: An empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interact. 24(6), 574–594 (2008). https://doi.org/10.1080/10447310802205776

    Article  Google Scholar 

  60. Michalco, J., Simonsen, J.G., Hornbæk, K.: An exploration of the relation between expectations and user experience. Int. J. Hum. Comput. Interact. 31(9), 603–617 (2015). https://doi.org/10.1080/10447318.2015.1065696

    Article  Google Scholar 

  61. Gould, D.J., Terrell, M.A., Fleming, J.: A usability study of users’ perceptions toward a multimedia computer-assisted learning tool for neuroanatomy. Anatom. Sci. Educ. 1(4), 175–183 (2008). https://doi.org/10.1002/ase.36

    Article  Google Scholar 

  62. Bringula, R.P., Basa, R.S.: Factors affecting faculty web portal usability. J. Educ. Technol. Soc. 14(4), 253–265 (2011)

    Google Scholar 

  63. Flavián, C., Guinalíu, M., Gurrea, R.: The role played by perceived usability, satisfaction and consumer trust on website loyalty. Inf. Manag. 43(1), 1–14 (2006). https://doi.org/10.1016/j.im.2005.01.002

    Article  Google Scholar 

  64. Sauro, J., Lewis, J.R.: Quantifying the User Experience: Practical Statistics for User Research. Morgan Kaufmann, Burlington (2016). https://doi.org/10.1007/s13398-014-0173-7.2

    Book  Google Scholar 

  65. Wu, Y., Li, Z., Wu, W., Zhou, M.: Response selection with topic clues for retrieval-based chatbots. Neurocomputing 316, 251–261 (2018). https://doi.org/10.1016/j.neucom.2018.07.073

    Article  Google Scholar 

  66. Jin, W., Zhirui, D.: Research on mobile learning model of college English based on WeChat platform. Eurasia J. Math. Sci. Technol. Educ. 13(8), 5847–5853 (2017). https://doi.org/10.12973/eurasia.2017.01034a

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China under Grant No. 61272205 and the Technology Innovation Special Projects of Hubei Province under Grant No. 2017ACA105. We would like to express our sincere and heartfelt thanks to Prof. Wenli Chen from the Nanyang Technological University and to the editors and reviewers of this paper.

Author information

Authors and Affiliations

  1. Central China Normal University, Wuhan, Hubei, China

    Qingtang Liu, Jingxiu Huang, Linjing Wu & Shen Ba

  2. Henan Normal University, Xinxiang, Henan, China

    Ke Zhu

Authors
  1. Qingtang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingxiu Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linjing Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ke Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shen Ba
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jingxiu Huang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

1.1 Usability evaluation questionnaire

Basic Information

   

Questions

Options

   

1

What is your gender?

Male

Female

   

2

What is your educational background?

Undergraduate

Graduate

   

System Usability Scale (SUS)

Statements

5-point scale

1

I think I would like to use the CBET as an out-of-class learning tool frequently.

1

2

3

4

5

2

I think the CBET is unnecessarily complex.

1

2

3

4

5

3

I thought the CBET was easy to use.

1

2

3

4

5

4

I think that I would need the support of a technical person to be able to use the CBET.

1

2

3

4

5

5

I found the various functions in the CBET were well integrated.

1

2

3

4

5

6

I thought there was too much inconsistency in the CBET.

1

2

3

4

5

7

I would imagine that most schoolmates would learn to use the CBET very quickly.

1

2

3

4

5

8

I found the CBET very awkward to use.

1

2

3

4

5

9

I felt very confident using the CBET.

1

2

3

4

5

10

I needed to learn a lot of things before I could get going with the CBET.

1

2

3

4

5

11

Overall, I would rate the user friendliness of the CBET as:

awful

poor

ok

good

excellent

Likely Recommendation

Question

Likely scale

How likely are you to recommend the CBET to a friend or schoolmate?

0

1

2

3

4

5

6

7

8

9

10

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Q., Huang, J., Wu, L. et al. CBET: design and evaluation of a domain-specific chatbot for mobile learning. Univ Access Inf Soc 19, 655–673 (2020). https://doi.org/10.1007/s10209-019-00666-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-019-00666-x

Keywords

Search

Navigation