Skip to main content
SpringerLink
Log in

Beyond engagement: an EEG-based methodology for assessing user’s confusion in an educational game

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

Confusion is an emotion, which may occur when the learner is confronting inconsistence between new knowledge and existing cognitive structure, or reasoning for solving the puzzle and problem. Although confusion is not pleasant, it is necessary for the learner to engage in understanding and deep learning. Consequently, confusion assessment has attracted increased interest in e-learning. However, current studies have targeted no further than engagement detection and measurement, while there is lack of studies in investigating cognitive and emotional aspects beyond engagement in the context of game-based learning. To quantify confused states in logic reasoning in game-based learning, we propose an EEG-based methodology for assessing the user’s confusion using the OpenBCI device with 8 channels. In the complicated context of game play, it is difficult, and sometimes impossible, to collect the ground truth of the data in real tasks. To solve this issue, this work leverages cross-task and cross-subject methods to build a classifier, that is, training on the data of one standardized cognitive test paradigm (Raven’s test) and testing on the data of real tasks in game play (Sokoban Game). It provides a new possibility to create a classifier based on a small dataset. We also employ the end-to-end algorithm of deep learning in machine learning. Results showed the feasibility of this proposal in the task variation of the classifier, with an accuracy of 91.04%. The proposed EEG-based methodology is suitable to analyze learners’ confusion on the long game-play duration and has a good adaption in real tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. All, A., Castellar, E.P.N., Looy, J.V.: Assessing the effectiveness of digital game-based learning: Best practices. Comput. Educ. 92–93, 90–103 (2016)

    Article  Google Scholar 

  2. Anderson, J.R.: Cognitive Psychology and its Implications. A Series of Books in Psychology, 6th edn. Worth Publishers, New York (2004)

    Google Scholar 

  3. Antoniades, A., Spyrou, L., Took, C.C., Sanei, S.: Deep learning for epileptic intracranial EEG data. In: 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1–6 (2016)

  4. Bashivan, P., Rish, I., Yeasin, M., Codella, N.: Learning representations from EEG with deep recurrent-convolutional neural networks. CoRR abs/1511.06448 (2015)

  5. Bradley, M.M., Lang, P.J.: Measuring emotion: The self-assessment manikin and the semantic differential. J. Behav. Therap. Exp. Psychiatry 25(1), 49 (1994)

    Article  Google Scholar 

  6. Chanel, G., Rebetez, C., Bétrancourt, M., Pun, T.: Emotion assessment from physiological signals for adaptation of game difficulty. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 41(6), 1052–1063 (2011)

    Article  Google Scholar 

  7. Clark, R.C., Mayer, R.E.: E-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning. Electronic Learning and the Science of Instruction, 3rd edn. Pfeiffer, San Francisco (2011)

    Book  Google Scholar 

  8. Clore, G.L., Huntsinger, J.R.: How emotions inform judgment and regulate thought. Trends Cogn. Sci. 11(9), 393–399 (2007)

    Article  Google Scholar 

  9. D’Mello, S., Graesser, A.: Dynamics of affective states during complex learning. Learn. Instr. 22(2), 145–157 (2012)

    Article  Google Scholar 

  10. D’Mello, S., Lehman, B., Pekrun, R., Graesser, A.: Confusion can be beneficial for learning. Learn. Instr. 29(X), 153–170 (2014)

    Article  Google Scholar 

  11. E-prime 2.0: http://www.psychology-software-tools.mybigcommerce.com/e-prime-2-0-professional/. Accessed 19 Nov 2018

  12. Ghali, R., Ouellet, S., Frasson, C.: Lewispace: an exploratory study with a machine learning model in an educational game. J. Educ. Train. Stud. 4(1), 192–201 (2016)

    Google Scholar 

  13. Ghergulescu, I., Muntean, C.H.: A novel sensor-based methodology for learner’s motivation analysis in game-based learning. Interact. Comput. 26(4), 305–320 (2014)

    Article  Google Scholar 

  14. Graesser, A.C., Lu, S., Olde, B.A., Cooper-Pye, E., Whitten, S.: Question asking and eye tracking during cognitive disequilibrium: comprehending illustrated texts on devices when the devices break down. Mem. Cogn. 33(7), 1235–1247 (2005). https://doi.org/10.3758/BF03193225

    Article  Google Scholar 

  15. Grimes, D., Tan, D.S., Hudson, S.E., Shenoy, P., Rao, R.P.: Feasibility and pragmatics of classifying working memory load with an electroencephalograph. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’08, pp. 835–844. ACM, New York (2008). https://doi.org/10.1145/1357054.1357187

  16. Hajinoroozi, M., Mao, Z., Jung, T.P., Lin, C.T., Huang, Y.: Eeg-based prediction of driver’s cognitive performance by deep convolutional neural network. Signal Process. Image Commun. 47, 549–555 (2016)

    Article  Google Scholar 

  17. Halpern, D.F., Millis, K., Graesser, A.C., Butler, H., Forsyth, C., Cai, Z.: Operation ARA: a computerized learning game that teaches critical thinking and scientific reasoning. Think. Skills Creat. 7(2), 93–100 (2012)

    Article  Google Scholar 

  18. Ifenthaler, D., Eseryel, D., Ge, X.: Assessment for Game-Based Learning, pp. 1–8. Springer, New York (2012)

    Book  Google Scholar 

  19. Jiao, Z., Gao, X., Wang, Y., Li, J., Xu, H.: Deep convolutional neural networks for mental load classification based on EEG data. Pattern Recogn 76, 582–595 (2018)

    Article  Google Scholar 

  20. Kapoor, A., Picard, R.W.: Multimodal affect recognition in learning environments. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, MULTIMEDIA ’05, pp. 677–682. ACM, New York (2005). https://doi.org/10.1145/1101149.1101300

  21. Kim, M.K., Kim, M., Oh, E., Kim, S.P.: A review on the computational methods for emotional state estimation from the human EEG. Comput. Math. Methods Med. 2013, 13 (2013). https://doi.org/10.1155/2013/573734

    MathSciNet  MATH  Google Scholar 

  22. Kirriemuir, J., Mcfarlane, A.: Literature review in games and learning. A NESTA Futurelab Research report - report 8 (2004). https://telearn.archives-ouvertes.fr/hal-00190453

  23. Klimesch, W.: Eeg alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Res. Rev. 29(2), 169–195 (1999). https://doi.org/10.1016/S0165-0173(98)00056-3

    Article  Google Scholar 

  24. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436 (2015)

    Article  Google Scholar 

  25. Lehman, B., D’Mello, S., Graesser, A.: Confusion and complex learning during interactions with computer learning environments. Internet High. Educ. 15(3), 184–194 (2012)

    Article  Google Scholar 

  26. Lehman, B., Matthews, M., D’Mello, S., Person, N.: What are you feeling? Investigating student affective states during expert human tutoring sessions. In: International Conference on Intelligent Tutoring Systems, pp. 50–59 (2008)

  27. Li, M., Lu, B.L.: Emotion classification based on gamma-band EEG. In: 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 1223–1226 (2009)

  28. Liu, Y.J., Yu, M., Zhao, G., Song, J., Ge, Y., Shi, Y.: Real-time movie-induced discrete emotion recognition from EEG signals. IEEE Trans. Affect. Comput. 9(4), 550–562 (2017)

    Article  Google Scholar 

  29. Moreno-Ger, P., Burgos, D., Martínez-Ortiz, I., Sierra, J.L., Fernández-Manjón, B.: Educational game design for online education. Comput. Hum. Behav. 24(6), 2530–2540 (2008). (Including the Special Issue: Electronic Games and Personalized eLearning Processes)

    Article  Google Scholar 

  30. Nie, D., Wang, X.W., Shi, L.C., Lu, B.L.: EEG-based emotion recognition during watching movies. In: 2011 5th International IEEE/EMBS Conference on Neural Engineering, pp. 667–670 (2011)

  31. Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010). https://doi.org/10.1109/TKDE.2009.191

    Article  Google Scholar 

  32. Pekrun, R., Linnenbrink-Garcia, L.: Academic Emotions and Student Engagement, pp. 259–282. Springer, Boston (2012)

    Google Scholar 

  33. Qian, M., Clark, K.R.: Game-based learning and 21st century skills: a review of recent research. Comput. Hum. Behav. 63, 50–58 (2016)

    Article  Google Scholar 

  34. Raven, J.: The raven’s progressive matrices: change and stability over culture and time. Cogn. Psychol. 41(1), 1–48 (2000)

    Article  MathSciNet  Google Scholar 

  35. Ren, Y., Wu, Y.: Convolutional deep belief networks for feature extraction of EEG signal. In: 2014 International Joint Conference on Neural Networks (IJCNN), pp. 2850–2853 (2014)

  36. Rob, B.K., Reilly, R., Picard, R.W.: External representation of learning process and domain knowledge: affective state as a determinate of its structure and function. In: Artificial Intelligence in Education Workshops, pp. 64–69 (2001)

  37. Saeed, S., Zyngier, D.: How motivation influences student engagement: a qualitative case study. J Educ Learn 1, 252–267 (2012)

    Article  Google Scholar 

  38. Schunk, D., Pintrich, P., Meece, J.: Motivation in Education: Theory, Research, and Applications. Pearson/Merrill Prentice Hall, Upper Saddle River (2008)

    Google Scholar 

  39. Sharma, N., Gedeon, T.: Objective measures, sensors and computational techniques for stress recognition and classification: a survey. Comput. Methods Progr. Biomed. 108(3), 1287–1301 (2012). https://doi.org/10.1016/j.cmpb.2012.07.003

    Article  Google Scholar 

  40. Silvia, P.J.: Confusion and interest: the role of knowledge emotions in aesthetic experience. Psychol. Aesthet. Creat. Arts 4(2), 75–80 (2010). https://doi.org/10.1037/a0017081

    Article  MathSciNet  Google Scholar 

  41. Sturm, I., Lapuschkin, S., Samek, W., Müller, K.R.: Interpretable deep neural networks for single-trial EEG classification. J. Neurosci. Methods 274, 141–145 (2016)

    Article  Google Scholar 

  42. Sun, X., Qian, C., Chen, Z., Wu, Z., Luo, B., Pan, G.: Remembered or forgotten?—an eeg-based computational prediction approach. PLOS ONE 11(12), 1–20 (2016)

    Google Scholar 

  43. Szafir, D., Mutlu, B.: Pay attention!: designing adaptive agents that monitor and improve user engagement. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’12, pp. 11–20. ACM, New York (2012)

  44. Tang, Z., Li, C., Sun, S.: Single-trial EEG classification of motor imagery using deep convolutional neural networks. Optik Int. J. Light Electron Opt. 130, 11–18 (2017)

    Article  Google Scholar 

  45. Wang, H., Li, Y., Hu, X., Yang, Y., Meng, Z., Chang, K.M.: Using EEG to improve massive open online courses feedback interaction. In: AI-ED Workshop Proceedings, pp. 59–66 (2013)

  46. Wang, X.W., Nie, D., Lu, B.L.: Emotional state classification from EEG data using machine learning approach. Neurocomputing 129(4), 94–106 (2014)

    Article  Google Scholar 

  47. Whitehill, J., Serpell, Z., Lin, Y.C., Foster, A., Movellan, J.R.: The faces of engagement: automatic recognition of student engagement from facial expressions. IEEE Trans. Affect. Comput. 5(1), 86–98 (2014)

    Article  Google Scholar 

  48. Xu, J., Zhong, B.: Review on portable EEG technology in educational research. Comput. Hum. Behav. 81, 340–349 (2018). https://doi.org/10.1016/j.chb.2017.12.037

    Article  Google Scholar 

  49. Xu, T., Zhou, Y., Wang, Z., Peng, Y.: Learning emotions eeg-based recognition and brain activity: a survey study on BCI for intelligent tutoring system. In: Procedia Computer Science. The 9th International Conference on Ambient Systems, Networks and Technologies (ANT 2018), vol. 130, pp. 376–382 (2018)

  50. Zhou, Y., Xu, T., Cai, Y., Wu, X., Dong, B.: Monitoring cognitive workload in online videos learning through an EEG-based brain-computer interface. Learning and Collaboration Technologies. Novel Learning Ecosystems, pp. 64–73. Springer, Cham (2017)

    Chapter  Google Scholar 

  51. Zhou, Y., Xu, T., Zhu, Z., Wang, Z.: Learning in doing: a model of design and assessment for using new interaction in educational game. In: Zaphiris, P., Ioannou, A. (eds.) Learning and Collaboration Technologies. Learning and Teaching, pp. 225–236. Springer, Cham (2018)

    Chapter  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (61703259 and 61702417) and the Natural Science Foundation of Shaanxi Province (2017JM6097).

Author information

Authors and Affiliations

  1. School of Education, Shaanxi Normal University, Xi’an, 710062, People’s Republic of China

    Yun Zhou

  2. School of Software, Northwestern Polytechnical University, Xi’an, 710072, People’s Republic of China

    Tao Xu, Shaoqi Li & Ruifeng Shi

Authors
  1. Yun Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tao Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shaoqi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ruifeng Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yun Zhou or Tao Xu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, Y., Xu, T., Li, S. et al. Beyond engagement: an EEG-based methodology for assessing user’s confusion in an educational game. Univ Access Inf Soc 18, 551–563 (2019). https://doi.org/10.1007/s10209-019-00678-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-019-00678-7

Keywords

Search

Navigation