Skip to main content
SpringerLink
Log in

Methodological Considerations for Understanding Students’ Problem Solving Processes and Affective Trajectories During Game-Based Learning: A Data Fusion Approach

  • Conference paper
  • First Online:
HCI in Games: Serious and Immersive Games (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12790))

Included in the following conference series:

  • 1620 Accesses

Abstract

This paper describes methodological considerations for fusing data sources in understanding both affective and problem solving states of students as they engage in computational thinking (CT) game-based learning. We provide both a theoretical and empirical rationale for using data including facial recognition and students’ logfile data to gain a more robust explanation of why students may experience emotions such as frustration during CT game-based learning activities. We showcase illustrative examples using data from individual learners to highlight the methodological approaches that we have taken. Finally, given the complexities of understanding constructs such as affect and problem solving, we provide a rationale for using a data fusion methodological approach.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Almeda, M.V., et al.: Modeling implicit computational thinking in zoombinis mudball wall puzzle gameplay. Paper submitted to the Technology, Mind, and Society conference, October, Washington D.C. (2019)

    Google Scholar 

  2. Alonso-Fernández, C., Calvo-Morata, A., Freire, M., Martinez-Ortiz, I., Fernández-Manjón, B.: Applications of data science to game learning analytics data: a systematic literature review. Comput. Educ. 141, 103612 (2019)

    Google Scholar 

  3. Alonso‐Fernández, C., Martínez‐Ortiz, I., Caballero, R., Freire, M., Fernández‐Manjón, B.: Predicting students’ knowledge after playing a serious game-based on learning analytics data: a case study. J. Comput. Assist. Learn. 36(3), 350–358 (2020)

    Google Scholar 

  4. Andres, J.M.L., Rodrigo, M.M.T., Baker, R.S., Paquette, L., Shute, V.J., Ventura, M.: Analyzing student action sequences and affect while playing physics playground. In: EDM (Workshops) (2015)

    Google Scholar 

  5. Asbell-Clarke, J., et al.: The development of students’ computational thinking practices in elementary-and middle-school classes using the learning game, zoombinis. Comput. Hum. Behav. 115, 106587 (2021)

    Google Scholar 

  6. Azevedo, R., Gašević, D.: Analyzing multimodal multichannel data about self- regulated learning with advanced learning technologies: issues and challenges (2019)

    Google Scholar 

  7. Barr, D., Harrison, J., Conery, L.: Computational thinking: a digital age skill for everyone. Learn. Leading Technol. 38(6), 20–23 (2011)

    Google Scholar 

  8. Barr, V., Stephenson, C.: Bringing computational thinking to K-12: what is involved and what is the role of the computer science education community? ACM Inroads 2(1), 48–54 (2011)

    Article  Google Scholar 

  9. Blikstein, P.: Multimodal learning analytics. In: Proceedings of the Third International Conference on Learning Analytics and Knowledge, pp. 102–106 (2013)

    Google Scholar 

  10. Bosch, N., D’mello, S.K., Ocumpaugh, J., Baker, R.S., Shute, V.: Using video to automatically detect learner affect in computer-enabled classrooms. ACM Trans. Interact. Intell. Syst. (TiiS) 6(2), 1–26 (2016)

    Google Scholar 

  11. Bosch, N., D’Mello, S.: The affective experience of novice computer programmers. Int. J. Artif. Intell. Educ. 27(1), 181–206 (2017)

    Article  Google Scholar 

  12. Cloude, E.B., Taub, M., Lester, J., Azevedo, R.: The role of achievement goal orientation on metacognitive process use in game-based learning. In: Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., Luckin, R. (eds.) AIED 2019. LNCS (LNAI), vol. 11626, pp. 36–40. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23207-8_7

    Chapter  Google Scholar 

  13. DeHaan, R.L.: Teaching creativity and inventive problem solving in science. CBE—Life Sci. Educ. 8(3), 172–181 (2009)

    Google Scholar 

  14. Dever, D.A., Wiedbusch, M., Azevedo, R.: Learners’ gaze behaviors and metacognitive judgments with an agent-based multimedia environment. In: Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., Luckin, R. (eds.) AIED 2019. LNCS (LNAI), vol. 11626, pp. 58–61. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23207-8_11

    Chapter  Google Scholar 

  15. Di Mitri, D., Schneider, J., Specht, M., Drachsler, H.: From signals to knowledge: a conceptual model for multimodal learning analytics. J. Comput. Assist. Learn. 34(4), 338–349 (2018)

    Article  Google Scholar 

  16. D’Mello, S., Graesser, A., et al.: Monitoring affective trajectories during complex learning. In: Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 29 (2007)

    Google Scholar 

  17. Emerson, A., Cloude, E.B., Azevedo, R., Lester, J.: Multimodal learning analytics for game-based learning. Br. J. Edu. Technol. 51(5), 1505–1526 (2020)

    Article  Google Scholar 

  18. Grafsgaard, J., Wiggins, J., Boyer, K.E., Wiebe, E., Lester, J.: Predicting learning and affect from multimodal data streams in task-oriented tutorial dialogue. In: Educational Data Mining 2014 (2014)

    Google Scholar 

  19. Isen, A.M.: Some ways in which positive affect influences decision making and problem solving. Handbook Emot. 3, 548–573 (2008)

    Google Scholar 

  20. Kang, J., Liu, M., Qu, W.: Using gameplay data to examine learning behavior patterns in a serious game. Comput. Hum. Behav. 72, 757–770 (2017)

    Article  Google Scholar 

  21. Krumm, A., Means, B., Bienkowski, M.: Learning Analytics Goes to School: A Collaborative Approach to Improving Education. Routledge (2018)

    Google Scholar 

  22. Lane, H.C., D’Mello, S.K.: Uses of physiological monitoring in intelligent learning environments: a review of research, evidence, and technologies. In: Parsons, T.D., Lin, L., Cockerham, D. (eds.) Mind, Brain and Technology. ECTII, pp. 67–86. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-02631-8_5

    Chapter  Google Scholar 

  23. Lee, J.Y., Donkers, J., Jarodzka, H., Van Merriënboer, J.J.: How prior knowledge affects problem solving performance in a medical simulation game: using game-logs and eye-tracking. Comput. Hum. Behav. 99, 268–277 (2019)

    Google Scholar 

  24. Loderer, K., Pekrun, R., Lester, J.C.: Beyond cold technology: a systematic review and meta-analysis on emotions in technology-based learning environments. Learn. Instruct. 70 101162 (2018)

    Google Scholar 

  25. Lu, J.J., Fletcher, G.H.: Thinking about computational thinking. In: Proceedings of the 40th ACM Technical Symposium on Computer Science Education, pp. 260–264 (2009)

    Google Scholar 

  26. Mayer, R.E.: Computer games in education. Ann. Rev. Psychol. 70, 531–549 (2019)

    Article  Google Scholar 

  27. Mohaghegh, D.M., McCauley, M.: Computational thinking: the skill set of the 21st century (2016)

    Google Scholar 

  28. Ochoa, X., Worsley, M.: Augmenting learning analytics with multimodal sensory data. J. Learn. Analytics 3(2), 213–219 (2016)

    Article  Google Scholar 

  29. Oviatt, S., Grafsgaard, J., Chen, L., Ochoa, X.: Multimodal learning analytics: assessing learners’ mental state during the process of learning. In: The Hand- book of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition-Volume 2, pp. 331–374 (2018)

    Google Scholar 

  30. Plass, J.L., Homer, B.D., Kinzer, C.K.: Foundations of game-based learning. Educ. Psychol. 50(4), 258–283 (2015)

    Article  Google Scholar 

  31. Qian, M., Clark, K.R.: Game-based learning and 21st century skills: a review of recent research. Comput. Hum. Behav. 63, 50–58 (2016)

    Article  Google Scholar 

  32. Rabiner, L., Juang, B.: An introduction to hidden markov models. IEEE ASSP Mag. 3(1), 4–16 (1986)

    Google Scholar 

  33. Rowe, E., Asbell-Clarke, J., Cunningham, K., Gasca, S.: Assessing implicit computational thinking in zoombinis gameplay: pizza pass, fleens & bubblewonder abyss. In: Extended Abstracts Publication of the Annual Symposium on Computer- Human Interaction in Play, pp. 195–200 (2017)

    Google Scholar 

  34. Sankaran, S.R., Bui, T.: Impact of learning strategies and motivation on performance: a study in web-based instruction. J. Instr. Psychol. 28(3), 191 (2001)

    Google Scholar 

  35. Sawyer, R., Rowe, J., Azevedo, R., Lester, J.: Filtered time series analyses of student problem solving behaviors in game-based learning. Int. Educ. Data Mining Soc. (2018)

    Google Scholar 

  36. Schneider, B., Blikstein, P.: Unraveling students’ interaction around a tangible interface using multimodal learning analytics. J. Educ. Data Mining 7(3), 89–116 (2015)

    Google Scholar 

  37. Schwarz, N.: Feelings-as-information theory. Handbook Theories Soc. Psychol. 1, 289–308 (2011)

    Google Scholar 

  38. Shute, V.J., Sun, C., Asbell-Clarke, J.: Demystifying computational thinking. Educ. Res. Rev. 22, 142–158 (2017)

    Article  Google Scholar 

  39. Simon, M.A.: Reconstructing mathematics pedagogy from a constructivist perspective. J. Res. Math. Educ. 26(2), 114–145 (1995)

    Google Scholar 

  40. Sutton, R.S., Barto, A.G., et al.: Introduction to Reinforcement Learning, vol. 135. MIT press Cambridge (1998)

    Google Scholar 

  41. Sztajn, P., Confrey, J., Wilson, P.H., Edgington, C.: Learning trajectory based instruction: toward a theory of teaching. Educ. Res. 41(5), 147–156 (2012)

    Article  Google Scholar 

  42. Taub, M., Azevedo, R.: Using sequence mining to analyze metacognitive monitoring and scientific inquiry based on levels of efficiency and emotions during game-based learning. J. Educ. Data Mining 10(3), 1–26 (2018)

    Google Scholar 

  43. Taub, M., Mudrick, N.V., Azevedo, R., Millar, G.C., Rowe, J., Lester, J.: Using multi-channel data with multi-level modeling to assess in-game performance during gameplay with crystal island. Comput. Hum. Behav. 76, 641–655 (2017)

    Article  Google Scholar 

  44. Taub, M., Sawyer, R., Smith, A., Rowe, J., Azevedo, R., Lester, J.: The agency effect: the impact of student agency on learning, emotions, and problem solving behaviors in a game-based learning environment. Comput. Educ. 147, 103781 (2020)

    Article  Google Scholar 

  45. Tissenbaum, M.: I see what you did there! divergent collaboration and learner transitions from unproductive to productive states in open-ended inquiry. Comput. Educ. 145, 103739 (2020)

    Article  Google Scholar 

  46. Tsai, M.J., Huang, L.J., Hou, H.T., Hsu, C.Y., Chiou, G.L.: Visual behavior, flow and achievement in game-based learning. Comput. Educ. 98, 115–129 (2016)

    Article  Google Scholar 

  47. Vlachopoulos, D., Makri, A.: The effect of games and simulations on higher education: a systematic literature review. Int. J. Educ. Technol. High. Educ. 14(1), 1–33 (2017). https://doi.org/10.1186/s41239-017-0062-1

    Article  Google Scholar 

  48. Wiltshire, T.J., Butner, J.E., Fiore, S.M.: Problem solving phase transitions during team collaboration. Cogn. Sci. 42(1), 129–167 (2018)

    Article  Google Scholar 

  49. Wine, J.: Test anxiety and direction of attention. Psychol. Bull. 76(2), 92 (1971)

    Article  Google Scholar 

  50. Wolfram: The facial expression recognition 2013 (fer-2013) dataset. The Wolfram Data Repository (2018)

    Google Scholar 

  51. Xu, Z., Woodruff, E.: Person-centered approach to explore learner’s emotionality in learning within a 3D narrative game. In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference, pp. 439–443 (2017)

    Google Scholar 

  52. Yadav, A., Hong, H., Stephenson, C.: Computational thinking for all: pedagogical approaches to embedding 21st century problem solving in k-12 classrooms. TechTrends 60(6), 565–568 (2016)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Florida, Gainesville, FL, 32611, USA

    Maya Israel & Tongxi Liu

  2. Florida State University, Tallahassee, FL, 32306-4453, USA

    Jewoong Moon & Fengfeng Ke

  3. TERC, Cambridge, MA, 02140, USA

    Ibrahim Dahlstrom-Hakki

Authors
  1. Maya Israel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tongxi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jewoong Moon
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fengfeng Ke
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ibrahim Dahlstrom-Hakki
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maya Israel .

Editor information

Editors and Affiliations

  1. DePaul University, Chicago, IL, USA

    Xiaowen Fang

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Israel, M., Liu, T., Moon, J., Ke, F., Dahlstrom-Hakki, I. (2021). Methodological Considerations for Understanding Students’ Problem Solving Processes and Affective Trajectories During Game-Based Learning: A Data Fusion Approach. In: Fang, X. (eds) HCI in Games: Serious and Immersive Games. HCII 2021. Lecture Notes in Computer Science(), vol 12790. Springer, Cham. https://doi.org/10.1007/978-3-030-77414-1_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77414-1_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77413-4

  • Online ISBN: 978-3-030-77414-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation