Skip to main content
SpringerLink
Log in

Eye to Eye: Gaze Patterns Predict Remote Collaborative Problem Solving Behaviors in Triads

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13355))

Included in the following conference series:

Abstract

We investigated the feasibility of using eye gaze to model collaborative problem solving (CPS) behaviors in 96 triads (N = 288) who used videoconferencing to remotely collaborate on an educational physics game. Trained human raters coded spoken utterances based on a theoretically grounded framework consisting of three CPS facets: constructing shared knowledge, negotiation/coordination, and maintaining team function. We then trained random forest classifiers to identify each of the CPS facets using eye gaze features pertaining to each individual (e.g., number of fixations) and/or shared across individuals (e.g., eye gaze distance between collaborators) in conjunction with information about the unfolding task context. We found that the individual gaze features outperformed the shared features, and together yielded between 6% to 8% improvements in classification accuracy above task-context baseline models, using a cross-validation scheme that generalized across teams. We discuss how our findings support CPS theories and the development of real-time intervention systems that provide actionable feedback to improve collaboration.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. OECD: PISA 2015 Results (Volume V): Collaborative Problem Solving. Organisation for Economic Co-operation and Development, Paris (2015)

    Google Scholar 

  2. Fiore, S.M., Graesser, A., Greiff, S.: Collaborative problem-solving education for the twenty-first-century workforce. Nat Hum Behav. 2, 367–369 (2018)

    Article  Google Scholar 

  3. Franken, E., et al.: Forced flexibility and remote working: opportunities and challenges in the new normal. J. Manag. Org. 1–19 (2021)

    Google Scholar 

  4. Kniffin, K.M., et al.: COVID-19 and the workplace: implications, issues, and insights for future research and action. Am. Psychol. 76, 63–77 (2021)

    Article  Google Scholar 

  5. Sottilare, R.A., Graesser, A.C., Hu, X., Sinatra, A.M.: Design Recommendations for Intelligent Tutoring Systems: Volume 6 - Team Tutoring. US Army Research Laboratory (2018)

    Google Scholar 

  6. Dyke, G., Adamson, D., Howley, I., Penstein Rosé, C.: Towards academically productive talk supported by conversational agents. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) Intelligent Tutoring Systems, vol. 7315, pp. 531–540. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-30950-2_69

    Chapter  Google Scholar 

  7. Schneider, B., Dowell, N., Thompson, K.: Collaboration analytics—current state and potential futures. J. Learn. Anal. 8, 1–12 (2021)

    Article  Google Scholar 

  8. Shockley, K., Richardson, D.C., Dale, R.: Conversation and coordinative structures. Top. Cogn. Sci. 1, 305–319 (2009)

    Article  Google Scholar 

  9. Duchowski, A.T.: Visual attention. In: Duchowski, A.T. (ed.) Eye Tracking Methodology: Theory and Practice, pp. 3–15. Springer, London (2003). https://doi.org/10.1007/978-1-84628-609-4_1

    Chapter  MATH  Google Scholar 

  10. Hao, J., Chen, L., Flor, M., Liu, L., von Davier, A.A.: CPS-Rater: automated sequential annotation for conversations in collaborative problem-solving activities. ETS Res. Rep. Ser. 2017, 1–9 (2017)

    Google Scholar 

  11. Pugh, S., Subburaj, S.K., Rao, A.R., Stewart, A., Andrews-Todd, J., D’Mello, S.: Say what? Automatic modeling of collaborative problem solving skills from student speech in the wild. In: Proceedings of the 14th Educational Data Mining Conference (2021)

    Google Scholar 

  12. Emara, M., Hutchins, N., Grover, S., Snyder, C., Biswas, G.: Examining student regulation of collaborative, computational, problem-solving processes in open-ended learning environments. J. Learn. Anal. 8, 49–74 (2021)

    Article  Google Scholar 

  13. Chopade, P., Edwards, D., Khan, S.M., Andrade, A., Pu, S.: CPSX: using AI-machine learning for mapping human-human interaction and measurement of CPS teamwork skills. In: 2019 IEEE International Symposium on Technologies for Homeland Security (HST), pp. 1–6 (2019)

    Google Scholar 

  14. Reilly, J.M., Schneider, B.: Predicting the quality of collaborative problem solving through linguistic analysis of discourse. International Educational Data Mining Society (2019)

    Google Scholar 

  15. Pugh, S.L., Rao, A., Stewart, A.E.B., D’Mello, S.K.: Do speech-based collaboration analytics generalize across task contexts? In: LAK 2022: 12th International Learning Analytics and Knowledge Conference, pp. 208–218. Association for Computing Machinery, NewYork, (2022)

    Google Scholar 

  16. Jermann, P., Sharma, K.: Gaze as a proxy for cognition and communication. In: 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT), pp. 152–154 (2018)

    Google Scholar 

  17. Olsen, J., Sharma, K., Rummel, N., Aleven, V.: Temporal analysis of multimodal data to predict collaborative learning outcomes. Br. J. Educ. Technol. 51 (2020)

    Google Scholar 

  18. Schneider, B., Sharma, K., Cuendet, S., Zufferey, G., Dillenbourg, P., Pea, R.: Leveraging mobile eye-trackers to capture joint visual attention in co-located collaborative learning groups. Int. J. Comput.-Support. Collab. Learn. 13(3), 241–261 (2018). https://doi.org/10.1007/s11412-018-9281-2

    Article  Google Scholar 

  19. Celepkolu, M., Boyer, K.E.: Predicting student performance based on eye gaze during collaborative problem solving. In: Proceedings of the Group Interaction Frontiers in Technology, pp. 1–8. Association for Computing Machinery, New York (2018)

    Google Scholar 

  20. Vrzakova, H., Amon, M.J., Stewart, A.E.B., D’Mello, S.K.: Dynamics of visual attention in multiparty collaborative problem solving using multidimensional recurrence quantification analysis. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland, UK, pp. 1–14. ACM (2019)

    Google Scholar 

  21. Sun, C., Shute, V.J., Stewart, A., Yonehiro, J., Duran, N., D’Mello, S.: Towards a generalized competency model of collaborative problem solving. Comput. Educ. 143, 103672 (2020)

    Article  Google Scholar 

  22. Subburaj, S.K., Stewart, A.E.B., Ramesh Rao, A., D’Mello, S.K.: Multimodal, multiparty modeling of collaborative problem solving performance. In: Proceedings of the 2020 International Conference on Multimodal Interaction, pp. 423–432. Association for Computing Machinery, New York (2020)

    Google Scholar 

  23. Stewart, A.E.B., Amon, M.J., Duran, N.D., D’Mello, S.K.: Beyond team makeup: diversity in teams predicts valued outcomes in computer-mediated collaborations. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–13. Association for Computing Machinery, New York (2020)

    Google Scholar 

  24. Shute, V.J., Ventura, M., Kim, Y.J.: Assessment and learning of qualitative physics in Newton’s playground. J. Educ. Res. 106, 423–430 (2013)

    Article  Google Scholar 

  25. Gwet, K.L.: Handbook of inter-rater reliability: the definitive guide to measuring the extent of agreement among raters‬ (2014)‬‬

    Google Scholar 

  26. Dalmaijer, E.S., Mathôt, S., Van der Stigchel, S.: PyGaze: An open-source, cross-platform toolbox for minimal-effort programming of eyetracking experiments. Behav. Res. Methods 46(4), 913–921 (2013). https://doi.org/10.3758/s13428-013-0422-2

    Article  Google Scholar 

  27. Richardson, D.C., Dale, R., Tomlinson, J.M.: Conversation, Gaze coordination, and beliefs about visual context. Cognit. Sci. 33, 1468–1482 (2009)

    Article  Google Scholar 

  28. Vrzakova, H., Amon, M.J., Rees, M., Faber, M., D’Mello, S.: Looking for a deal? Visual social attention during negotiations via mixed media videoconferencing. Proc. ACM Hum.-Comput. Interact. 4, 260:1–260:35 (2021)

    Google Scholar 

  29. Westlund, J.K., D’Mello, S.K., Olney, A.M.: Motion tracker: camera-based monitoring of bodily movements using motion Silhouettes. PLoS ONE 10, e0130293 (2015)

    Article  Google Scholar 

  30. Robin, X., et al.: pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinform. 12, 77 (2011)

    Article  Google Scholar 

  31. Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. in: advances in neural information processing systems. Curran Associates, Inc. (2017)

    Google Scholar 

  32. Pöysä-Tarhonen, J., Awwal, N., Häkkinen, P., Otieno, S.: Joint attention behaviour in remote collaborative problem solving: exploring different attentional levels in dyadic interaction. Res. Pract. Technol. Enhanc. Learn. 16(1), 1–24 (2021). https://doi.org/10.1186/s41039-021-00160-0

    Article  Google Scholar 

  33. Peacock, C.E., et al.: Gaze dynamics are sensitive to target orienting for working memory encoding in virtual reality. J. Vis. 22, 2 (2022)

    Article  Google Scholar 

  34. Stewart, A.E.B., Keirn, Z., D’Mello, S.K.: Multimodal modeling of collaborative problem-solving facets in triads. User Model. User-Adap. Inter. 31(4), 713–751 (2021). https://doi.org/10.1007/s11257-021-09290-y

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by the NSF National AI Institute for Student-AI Teaming (iSAT) (DRL 2019805) and NSF DUE 1745442/1660877. The opinions expressed are those of the authors and do not represent views of the NSF.

Author information

Authors and Affiliations

  1. University of Colorado Boulder, Boulder, CO, 80309, USA

    Angelina Abitino, Samuel L. Pugh, Candace E. Peacock & Sidney K. D’Mello

Authors
  1. Angelina Abitino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Samuel L. Pugh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Candace E. Peacock
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sidney K. D’Mello
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Angelina Abitino .

Editor information

Editors and Affiliations

  1. Ateneo De Manila University, Quezon, Philippines

    Maria Mercedes Rodrigo

  2. Department of Computer Science, North Carolina State University, Raleigh, NC, USA

    Noburu Matsuda

  3. Durham University, Durham, UK

    Alexandra I. Cristea

  4. University of Leeds, Leeds, UK

    Vania Dimitrova

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Abitino, A., Pugh, S.L., Peacock, C.E., D’Mello, S.K. (2022). Eye to Eye: Gaze Patterns Predict Remote Collaborative Problem Solving Behaviors in Triads. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2022. Lecture Notes in Computer Science, vol 13355. Springer, Cham. https://doi.org/10.1007/978-3-031-11644-5_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-11644-5_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-11643-8

  • Online ISBN: 978-3-031-11644-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation