Skip to main content
SpringerLink
Log in

Investigating Navigational Behavior Patterns of Students Across At-Risk Categories Within an Open-Ended Serious Game

  • Original research
  • Published:
Technology, Knowledge and Learning Aims and scope Submit manuscript

Abstract

An open-ended serious game can engage students’ scientific problem-solving processes. However, understanding how students learn higher-order thinking skills through solving a problem in an open-ended game system is a challenge. The complex game systems may make learning more difficult for students with different characteristics such as an at-risk label. Recent research stresses the importance of using gameplay data to better understand diverse individuals’ learning behaviors and performances in the context of serious games. We analyzed gameplay data of a serious game called Alien Rescue to identify navigation behavior patterns between at-risk and non-at-risk middle school students. Particularly, we incorporated the combination of lag sequential analysis and sequential pattern mining in statistical analyses. The results revealed that the non-at-risk and at-risk students used problem-solving strategies differently when they navigated the environment. The findings using this integrated method confirmed that additional support was needed for at-risk students in order for them to develop contextual and procedural knowledge for problem-solving in the game environment. The findings provide methodological guidelines for researchers considering a sequential analysis as well as offer practical guidelines for game designers to consider when designing serious games with a complex problem so as to help at-risk students.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Agrawal, R., & Srikant, R. (1995). Mining sequential patterns. In Proceedings of the eleventh IEEE international conference on data engineering (ICDE) (pp. 3–14). Taipei, Taiwan.

  • Anderson, J. R. (1980). Cognitive psychology and its implications. New York: W. H. Freeman and Company.

    Google Scholar 

  • Anderson, R. (2002). Reforming science teaching: What research says about inquiry. Journal of Science Teacher Education, 13, 1–2.

    Google Scholar 

  • Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis (2nd ed.). New York, NY: Cambridge University Press.

    Google Scholar 

  • Barab, S. A., Sadler, T., Heiselt, C., Hickey, D., & Zuiker, S. (2007). Relating narrative, inquiry, and inscriptions: A framework for socioscientific inquiry. Journal of Science Education and Technology, 16(1), 59–82.

    Google Scholar 

  • Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of serious games: An overview. Advances in Human-Computer Interaction, 2013, 11. https://doi.org/10.1155/2013/136864.

    Article  Google Scholar 

  • Bera, S., & Liu, M. (2006). Cognitive tools, individual differences, and group processing as mediating factors in a hypermedia environment. Computers in Human Behavior, 22(2), 295–319. https://doi.org/10.1016/j.chb.2004.05.001.

    Article  Google Scholar 

  • Bland, J. M., & Altman, D. G. (1995). Multiple significance tests: The Bonferroni method. British Medical Journal, 310, 170. https://doi.org/10.1136/bmj.310.6973.170.

    Article  Google Scholar 

  • Bogard, T., Liu, M., & Chiang, Y. H. (2013). Thresholds of knowledge development in complex problem solving: A multiple-case study of advanced learners’ cognitive processes. Educational Technology Research and Development, 61(3), 465–503. https://doi.org/10.1007/s11423-013-9295-4.

    Article  Google Scholar 

  • Bos, B. (2007). The effect of Texas Instrument interactive instructional environment on the mathematical achievement of eleventh grade low achieving students. Journal of Educational Computing Research, 37(4), 350–368.

    Google Scholar 

  • Brush, T., & Saye, J. (2000). Implementation and evaluation of a student-centered learning unit: A case study. Educational Technology Research and Development, 48(3), 70–100.

    Google Scholar 

  • Chen, Z., & Klahr, D. (1999). All other things being equal: Children’s acquisition of the control of variables strategy. Child Development, 70, 1098–1120.

    Google Scholar 

  • Chi, M. T. H., & Glaser, R. (1985). Problem solving ability. In R. Sternberg (Ed.), Human abilities: An information processing approach (pp. 227–250). San Francisco: Freeman.

    Google Scholar 

  • Chung, G. K., & Baker, E. L. (2003). An exploratory study to examine the feasibility of measuring problem-solving processes using a click-through interface. The Journal of Technology, Learning and Assessment. Retrieved from https://ejournals.bc.edu/index.php/jtla/article/view/1662.

  • Clark, D. B., Martinez-Garza, M. M., Biswas, G., Luecht, R. M., & Sengupta, P. (2012). Driving assessment of students’ explanations in game dialog using computer-adaptive testing and hidden Markov Modeling. In D. Ifenthaler, D. Eseryel, & G. Xun (Eds.), Game-based learning: Foundations, innovations, and perspectives (pp. 173–199). New York: Springer.

    Google Scholar 

  • Clarke-Midura, J., Dede, C., & Norton, J. (2011). The road ahead for state assessments. Policy Analysis for California Education and Rennie Center for Educational Research & Policy. Cambridge, MA: Rennie Center for Educational Research & Policy.

  • Cohen, L., Manion, L., & Morrison, K. (2013). Research methods in education (7th ed.). London: Routledge.

    Google Scholar 

  • Darling-Hammond, L., Zielezinski, M. B., & Goldman, S. (2014). Using technology to support at-risk students’ learning. Stanford, CA: Stanford Center for Opportunity Policy in Education.

    Google Scholar 

  • Djaouti, D., Alvarez, J., Jessel, J. P., & Rampnoux, O. (2011). Origins of serious games. In M. Ma, A. Oikonomou, & L. C. Jain (Eds.), Serious games and edutainment applications (pp. 25–43). London: Springer. https://doi.org/10.1007/978-1-4471-2161-9_3.

    Chapter  Google Scholar 

  • Esmaeili, M., & Gabor, F. (2010). Finding sequential patterns from large sequence data. International Journal of Computer Science Issues, 7(1), 43–46.

    Google Scholar 

  • Fournier-Viger, P., Lin, J. C. W., Kiran, R. U., Koh, Y. S., & Thomas, R. (2017). A survey of sequential pattern mining. Data Science and Pattern Recognition, 1(1), 54–77.

    Google Scholar 

  • Gee, J. P. (2003). What video games have to teach us about learning and literacy (2nd ed.). New York: Palgrave/Macmillan.

    Google Scholar 

  • Gijbels, D., Dochy, F., Van den Bossche, P., & Segers, M. (2005). Effects of problem-based learning: A meta-analysis from the angle of assessment. Review of Educational Research, 75(1), 27–61. https://doi.org/10.3102/00346543075001027.

    Article  Google Scholar 

  • Giorgino, T. (2009). Computing and visualizing dynamic time warping alignments in R: The dtw package. Journal of Statistical Software, 31(7), 1–24.

    Google Scholar 

  • Gobert, J. D., Kim, Y. J., Sao Pedro, M. A., Kennedy, M., & Betts, C. G. (2015). Using educational data mining to assess students’ skills at designing and conducting experiments within a complex systems microworld. Thinking Skills and Creativity, 18, 81–90.

    Google Scholar 

  • Gobert, J. D., Sao Pedro, M., Raziuddin, J., & Baker, R. S. (2013). From log files to assessment metrics: Measuring students’ science inquiry skills using educational data mining. Journal of the Learning Sciences, 22(4), 521–563. https://doi.org/10.1080/10508406.2013.837391.

    Article  Google Scholar 

  • Gott, R., Duggan, S., & Roberts, R. (2008). Concepts of evidence. Durham: School of Education, University of Durham.

    Google Scholar 

  • Gott, R., & Murphy, P. (1987). Assessing investigation at ages 13 and 15: Assessment of performance unit science report for teachers: 9. London: Department of Education and Science.

    Google Scholar 

  • Harpstead, E., MacLellan, C. J., Aleven, V., & Myers, B. A. (2015). Replay analysis in open-ended educational games. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement (pp. 381–399). Zurich: Springer. https://doi.org/10.1007/978-3-319-05834-4.

    Chapter  Google Scholar 

  • Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266.

    Google Scholar 

  • Hmelo-Silver, C. E., & Azevedo, R. (2006). Understanding complex systems: Some core challenges. Journal of the Learning Sciences, 15, 53–61.

    Google Scholar 

  • Hou, H. T. (2012). Exploring the behavioral patterns of learners in an educational massively multiple online role-playing game (MMORPG). Computers & Education, 58(4), 1225–1233.

    Google Scholar 

  • Hou, H. T. (2015). Integrating cluster and sequential analysis to explore learners’ flow and behavioral patterns in a simulation game with situated-learning context for science courses: A video-based process exploration. Computers in Human Behavior, 48, 424–435.

    Google Scholar 

  • Jonassen, D. H. (2004). Learning to solve problems: An instructional design guide. San Francisco: Pfeiffer.

    Google Scholar 

  • Kamarainen, A. M., Metcalf, S., Grotzer, T., Browne, A., Mazzuca, D., Tutwiler, M. S., et al. (2013). EcoMOBILE: Integrating augmented reality and probeware with environmental education field trips. Computers & Education, 68, 545–556. https://doi.org/10.1016/j.compedu.2013.02.018.

    Article  Google Scholar 

  • Kang, J., Liu, M., & Qu, W. (2017). Using gameplay data to examine learning behavior patterns in a serious game. Computers in Human Behavior, 72, 757–770. https://doi.org/10.1016/j.chb.2016.09.062.

    Article  Google Scholar 

  • Kang, J., An, D., Yan, L., & Liu, M. (2019). Collaborative problem-solving process in a science serious game: Exploring group action similarity trajectory. In C. F. Lynch, A. Merceron, M. Desmarais, & R. Nkambou (Eds.), Proceedings of the 12th international conference on educational data mining (pp. 336–341). International Data Mining Society.

  • Kinnebrew, J. S., & Biswas, G. (2012). Identifying learning behaviors by contextualizing differential sequence mining with action features and performance evolution. In K. Yacef, O. Zaïane, H. Hershkovitz, & J. Stamper (Eds.), Proceedings of the 5th international conference on educational data mining (pp. 57–64). International Data Mining Society.

  • Lamb, R. L., Annetta, L., Firestone, J., & Etopio, E. (2018). A meta-analysis with examination of moderators of student cognition, affect, and learning outcomes while using serious educational games, serious games, and simulations. Computers in Human Behavior, 80, 158–167. https://doi.org/10.1016/j.chb.2017.10.040.

    Article  Google Scholar 

  • Lederman, J. S., Lederman, N. G., Bartos, S. A., Bartels, S. L., Meyer, A. A., & Schwartz, R. S. (2014). Meaningful assessment of learners’ understandings about scientific inquiry—The views about scientific inquiry (VASI) questionnaire. Journal of Research in Science Teaching, 51(1), 65–83.

    Google Scholar 

  • Lin, J., Keogh, E., Wei, L., & Lonardi, S. (2007). Experiencing SAX: A novel symbolic representation of time series. Data Mining and Knowledge Discovery, 15(2), 107–144.

    Google Scholar 

  • Linek, S. B., Öttl, G., & Albert, D. (2010). Non-invasive data tracking in educational games: Combination of logfiles and natural language processing. In L. G. Chova, D. M. Belenguer (Eds.), INTED 2010: International technology, education and development conference, Spain, Valenica.

  • Liu, M., & Bera, S. (2005). An analysis of cognitive tool use patterns in a hypermedia learning environment. Educational Technology Research and Development, 53(1), 5–21. https://doi.org/10.1007/BF02504854.

    Article  Google Scholar 

  • Liu, M., Horton, L. R., Corliss, S. B., Svinicki, M. D., Bogard, T., Kim, J., et al. (2009). Students’ problem solving as mediated by their cognitive tool use: A study of tool use patterns. Journal of Educational Computing Research, 40(1), 111–139.

    Google Scholar 

  • Liu, M., Horton, L., Olmanson, J., & Toprac, P. (2011). A Study of learning and motivation in a new media enriched environment for middle school science. Educational Technology Research and Development, 59(2), 249–266. https://doi.org/10.1007/s11423-011-9192-7.

    Article  Google Scholar 

  • Liu, M., Kang, J., Lee, J., Winzeler, E., & Liu, S. (2015). Examining through visualization what tools learners access as they play a serious game for middle school science. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement (pp. 181–208). Switzerland: Springer. https://doi.org/10.1007/978-3-319-05834-4.

    Chapter  Google Scholar 

  • Loh, C. S. (2012). Information trails: In-process assessment of game-based learning. In D. Ifenthaler, D. Eseryel, & X. Ge (Eds.), Assessment in game-based learning: Foundations, innovations, and perspectives (pp. 123–144). New York: Springer. https://doi.org/10.1007/978-1-4614-3546-4.

    Chapter  Google Scholar 

  • Loh, C. S., & Sheng, Y. (2014). Maximum similarity index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games. Computers in Human Behavior, 39, 322–330.

    Google Scholar 

  • Loh, C. S., Sheng, Y., & Ifenthaler, D. (2015). Serious games analytics: Theoretical framework. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement (pp. 3–29). Zurich: Springer. https://doi.org/10.1007/978-3-319-05834-4.

    Chapter  Google Scholar 

  • Martinez, S. L. M., & Rury, J. L. (2012). From “Culturally Deprived” to “At Risk”: The politics of popular expression and educational inequality in the United States: 1960–1985. Teachers College Record, 114, 1–31.

    Google Scholar 

  • Mayer, R. E. (2008). Learning and instruction. Upper Saddle River, NJ: Merrill Prentice Hall.

    Google Scholar 

  • National Research Council. (1996). National science education standard. Washington, DC: The National Academies Press.

    Google Scholar 

  • Pei, J., Han, J., & Wang, W. (2007). Constraint-based sequential pattern mining: The pattern-growth methods. Journal of Intelligent Information Systems, 28(2), 133–160.

    Google Scholar 

  • Perera, D., Kay, J., Koprinska, I., Yacef, K., & Zaïane, O. R. (2008). Clustering and sequential pattern mining of online collaborative learning data. IEEE Transactions on Knowledge and Data Engineering, 21(6), 759–772.

    Google Scholar 

  • Pohl, M., Wallner, G., & Kriglstein, S. (2016). Using lag-sequential analysis for understanding interaction sequences in visualizations. International Journal of Human-Computer Studies, 96, 54–66. https://doi.org/10.1016/j.ijhcs.2016.07.006.

    Article  Google Scholar 

  • Ponticell, J. (2001). Making school more rewarding: At-risk students’ perspectives on teaching and learning. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA.

  • Quellmalz, E., Timms, M., & Schneider, S. (2009). Assessment of student learning in science simulations and games. In Proceedings of the workshop on learning science: Computer games, simulations, and education. Washington, DC: National Academy of Sciences.

  • Rowe, E., Asbell-Clarke, J., & Baker, R. (2015). Serious game analytics to measure implicit science learning. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics (pp. 343–360). Berlin: Springer.

    Google Scholar 

  • Samsonov, P., Pedersen, S., & Hill, C. L. (2006). Using problem-based learning software with at-risk students: A case study. Computers in the Schools, 23(1), 111–124.

    Google Scholar 

  • Sawyer, B., & Rejeski, D. (2002). Serious games: Improving public policy through game-based learning and simulation. Washington. DC: Woodrow Wilson International Center for Scholars.

    Google Scholar 

  • Schmidt, R. A., & Lee, T. (2011). Motor control and learning: A behavioral emphasis (5th ed.). Champaign, IL: Human Kinetics.

    Google Scholar 

  • Schunk, D. H. (2016). Learning theories: An educational perspective. Upper Saddle River, NJ: Pearson.

    Google Scholar 

  • Sifa, R., Drachen, A., & Bauckhage, C. (2018). Profiling in games: Understanding behavior from telemetry. In K. Lakkaraju, G. Sukthankar, & R. T. Wigand (Eds.), Social interactions in virtual worlds: An interdisciplinary perspective (pp. 337–374). Cambridge: Cambridge University Press.

    Google Scholar 

  • Simons, K., & Klein, J. (2007). The impact of scaffolding and student achievement levels in a problem-based learning environment. Instructional Science, 35(1), 41–72. https://doi.org/10.1007/s11251-006-9002-5.

    Article  Google Scholar 

  • Smith, S. P., Blackmore, K., & Nesbitt, K. (2015). A meta-analysis of data collection in serious games research. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement (pp. 31–55). Zurich: Springer.

    Google Scholar 

  • Spring, F., & Pellegrino, J. W. (2011). The challenge of assessing learning in open games: HORTUS as a case study. In Proceedings of the 8th games+learning+society conference—GLS 8.0 (pp. 209–217).

  • Squire, K. (2008). Open-ended video games: A model for developing learning for the interactive age. In K. Salen (Ed.), The ecology of games: Connecting youth, games, and learning (pp. 167–198). Cambridge, MA: The MIT Press. https://doi.org/10.1162/dmal.9780262693646.167.

    Chapter  Google Scholar 

  • Texas Education Agency. (2017). State compensatory education. Retrieved from https://tea.texas.gov/finance-and-grants/financial-compliance/statecompensatory-education.

  • van Barneveld, A., Arnold, K. E., & Campbell, J. P. (2012). Analytics in higher education: Establishing a common language. EDUCAUSE Learning Initiative.

  • Wallner, G., & Kriglstein, S. (2013). Visualization-based analysis of gameplay data—A review of literature. Entertainment Computing, 4(3), 143–155. https://doi.org/10.1016/j.entcom.2013.02.002.

    Article  Google Scholar 

  • Wallner, G., & Kriglstein, S. (2015). Comparative visualization of player behavior for serious game analytics. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement (pp. 159–179). Zurich: Springer. https://doi.org/10.1007/978-3-319-05834-4.

    Chapter  Google Scholar 

  • Wecker, C., Rachel, A., Heran-Dorr, E., Waltner, C., Wiesner, H., & Fischer, F. (2013). Presenting theoretical ideas prior to inquiry activities fosters theory-level knowledge. Journal of Research in Science Teaching, 50(10), 1180–1206.

    Google Scholar 

  • Welch, W. W., Klopfer, L. E., Aikenhead, G. S., & Robinson, J. (1981). The role of inquiry in science education: Analysis and recommendations. Science Education, 65(1), 33–50.

    Google Scholar 

  • Zaki, M. J. (2000). Scalable algorithms for association mining. IEEE Transactions on Knowledge and Data Engineering, 12(3), 372–390. https://doi.org/10.1109/69.846291.

    Article  Google Scholar 

  • Zaki, M. J. (2001). SPADE: An efficient algorithm for mining frequent sequences. Machine Learning, 42(1–2), 31–60. https://doi.org/10.1023/A:1007652502315.

    Article  Google Scholar 

  • Zhou, M., Xu, Y., Nesbit, J. C., & Winne, P. H. (2010). Sequential pattern analysis of learning logs: Methodology and applications. In C. Romero, et al. (Eds.), Handbook of educational data mining (pp. 107–121). Boca Raton: Chapman & Hall/CRC Press.

    Google Scholar 

  • Zyda, M. (2005). From visual simulation to virtual reality to games. Computer, 38(9), 25–32. https://doi.org/10.1109/mc.2005.297.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Instructional Technology and Learning Sciences, Utah State University, 2830 Old Main Hill, Logan, UT, 84322-2830, USA

    Jina Kang

  2. Learning Technologies Program, Department of Curriculum and Instruction, The University of Texas at Austin, 1912 Speedway Stop D5700, Austin, TX, 78712-1293, USA

    Min Liu

Authors
  1. Jina Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Min Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jina Kang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kang, J., Liu, M. Investigating Navigational Behavior Patterns of Students Across At-Risk Categories Within an Open-Ended Serious Game. Tech Know Learn 27, 183–205 (2022). https://doi.org/10.1007/s10758-020-09462-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10758-020-09462-6

Keywords

Search

Navigation