Skip to main content
SpringerLink
Log in

Face Forward: Detecting Mind Wandering from Video During Narrative Film Comprehension

  • Conference paper
  • First Online:
Book cover Artificial Intelligence in Education (AIED 2017)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10331))

Included in the following conference series:

Abstract

Attention is key to effective learning, but mind wandering, a phenomenon in which attention shifts from task-related processing to task-unrelated thoughts, is pervasive across learning tasks. Therefore, intelligent learning environments should benefit from mechanisms to detect and respond to attentional lapses, such as mind wandering. As a step in this direction, we report the development and validation of the first student-independent facial feature-based mind wandering detector. We collected training data in a lab study where participants self-reported when they caught themselves mind wandering over the course of completing a 32.5 min narrative film comprehension task. We used computer vision techniques to extract facial features and bodily movements from videos. Using supervised learning methods, we were able to detect a mind wandering with an F1 score of .390, which reflected a 31% improvement over a chance model. We discuss how our mind wandering detector can be used to adapt the learning experience, particularly for online learning contexts.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    A preliminary two-page version of this paper was presented as an Extended Abstract Poster at the 24th Conference on User Modeling, Adaptation and Personalization. The present paper describes the methods in more detail, updated results, and expanded analyses not included in the preliminary paper.

References

  1. D’Mello, S.K.: Giving eyesight to the blind: towards attention-aware AIED. Int. J. Artif. Intell. Educ. 26(2), 645–659 (2016)

    Article  Google Scholar 

  2. Killingsworth, M.A., Gilbert, D.T.: A wandering mind is an unhappy mind. Science 330, 932 (2010)

    Article  Google Scholar 

  3. Bixler, R., D’Mello, S.: Toward fully automated person-independent detection of mind wandering. In: Dimitrova, V., Kuflik, T., Chin, D., Ricci, F., Dolog, P., Houben, G.-J. (eds.) UMAP 2014. LNCS, vol. 8538, pp. 37–48. Springer, Cham (2014). doi:10.1007/978-3-319-08786-3_4

    Google Scholar 

  4. Kopp, K., Mills, C., D’Mello, S.K.: Mind wandering during film comprehension: the role of prior knowledge and situational interest. Psychon. Bull. Rev. 23, 842–848 (2015)

    Article  Google Scholar 

  5. Franklin, M.S., Smallwood, J., Schooler, J.W.: Catching the mind in flight: using behavioral indices to detect mindless reading in real time. Psychon. Bull. Rev. 18, 992–997 (2011)

    Article  Google Scholar 

  6. Szpunar, K.K., Moulton, S.T., Schacter, D.L.: Mind wandering and education: from the classroom to online learning. Front. Psychol. 4, 495 (2013)

    Article  Google Scholar 

  7. Randall, J.G., Oswald, F.L., Beier, M.E.: Mind-wandering, cognition, and performance: a theory-driven meta-analysis of attention regulation. Psychol. Bull. 140, 1411 (2014)

    Article  Google Scholar 

  8. Bixler, R., D’Mello, S.: Automatic gaze-based detection of mind wandering with metacognitive awareness. In: Ricci, F., Bontcheva, K., Conlan, O., Lawless, S. (eds.) UMAP 2015. LNCS, vol. 9146, pp. 31–43. Springer, Cham (2015). doi:10.1007/978-3-319-20267-9_3

    Chapter  Google Scholar 

  9. Blanchard, N., Bixler, R., Joyce, T., D’Mello, S.: Automated physiological-based detection of mind wandering during learning. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 55–60. Springer, Cham (2014). doi:10.1007/978-3-319-07221-0_7

    Chapter  Google Scholar 

  10. Pham, P., Wang, J.: AttentiveLearner: improving mobile MOOC learning via implicit heart rate tracking. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (eds.) AIED 2015. LNCS, vol. 9112, pp. 367–376. Springer, Cham (2015). doi:10.1007/978-3-319-19773-9_37

    Chapter  Google Scholar 

  11. Mills, C., Bixler, R., Wang, X., D’Mello, S.K.: Automatic gaze-based detection of mind wandering during film viewing. In: Proceedings of the 9th International Conference on Educational Data Mining. International Educational Data Mining Society, Raleigh (2016)

    Google Scholar 

  12. Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39–58 (2009)

    Article  Google Scholar 

  13. Faber, M., Bixler, R., D’Mello, S.K.: An automated behavioral measure of mind wandering during computerized reading. Behav. Res. Methods 1–17 (2017)

    Google Scholar 

  14. Reichle, E.D., Reineberg, A.E., Schooler, J.W.: Eye movements during mindless reading. Psychol. Sci. 21, 1300–1310 (2010)

    Article  Google Scholar 

  15. Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., Bartlett, M.: The computer expression recognition toolbox (CERT). In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), pp. 298–305. IEEE (2011)

    Google Scholar 

  16. Ekman, P., Friesen, W.V.: Facial action coding system (1977)

    Google Scholar 

  17. Westlund, J.K., D’Mello, S.K., Olney, A.M.: Motion tracker: camera-based monitoring of bodily movements using motion silhouettes. PLoS ONE 10, e0130293 (2015)

    Article  Google Scholar 

  18. Dixon, W.J., Yuen, K.K.: Trimming and winsorization: a review. Statistische Hefte 15, 157–170 (1974)

    Article  MATH  MathSciNet  Google Scholar 

  19. Allison, P.D.: Multiple Regression: A Primer. Pine Forge Press, Thousand Oaks (1999)

    Google Scholar 

  20. Kononenko, I.: Estimating attributes: analysis and extensions of RELIEF. In: Bergadano, F., Raedt, L. (eds.) ECML 1994. LNCS, vol. 784, pp. 171–182. Springer, Heidelberg (1994). doi:10.1007/3-540-57868-4_57

    Chapter  Google Scholar 

  21. Holmes, G., Donkin, A., Witten, I.H.: Weka: a machine learning workbench. In: Proceedings of the 1994 Second Australian and New Zealand Conference on Intelligent Information Systems, pp. 357–361. IEEE (1994)

    Google Scholar 

  22. Platt, J., et al.: Sequential minimal optimization: a fast algorithm for training support vector machines (1998)

    Google Scholar 

  23. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)

    MATH  Google Scholar 

  24. Cohen, J.: Statistical Power Analysis for the Behavioral Sciences, 2nd edn. Lawrence Erlbaum, Hillsdale (1988)

    MATH  Google Scholar 

  25. Szpunar, K.K., Khan, N.Y., Schacter, D.L.: Interpolated memory tests reduce mind wandering and improve learning of online lectures. Proc. Natl. Acad. Sci. 110, 6313–6317 (2013)

    Article  Google Scholar 

  26. Moss, J., Schunn, C.D., Schneider, W., McNamara, D.S.: The nature of mind wandering during reading varies with the cognitive control demands of the reading strategy. Brain Res. 1539, 48–60 (2013)

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by the National Science Foundation (NSF) (DRL 1235958 and IIS 1523091). Any opinions, findings and conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the NSF.

Author information

Authors and Affiliations

  1. University of Notre Dame, Notre Dame, IN, USA

    Angela Stewart, Patrick Donnelly & Sidney D’Mello

  2. University of Illinois at Urbana-Champaign, Urbana, IL, USA

    Nigel Bosch

  3. Massachusetts Institute of Technology, Cambridge, MA, USA

    Huili Chen

Authors
  1. Angela Stewart
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nigel Bosch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huili Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Patrick Donnelly
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sidney D’Mello
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Angela Stewart or Sidney D’Mello .

Editor information

Editors and Affiliations

  1. Human-Centered Multimedia, University of Augsburg, Augsburg, Germany

    Elisabeth André

  2. University of Pennsylvania, Philadelphia, Pennsylvania, USA

    Ryan Baker

  3. Department of Psychology, University of Memphis, Memphis, Tennessee, USA

    Xiangen Hu

  4. Ateneo de Manila University, Quezon City, Philippines

    Ma. Mercedes T. Rodrigo

  5. University of Sussex, Brighton, United Kingdom

    Benedict du Boulay

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Stewart, A., Bosch, N., Chen, H., Donnelly, P., D’Mello, S. (2017). Face Forward: Detecting Mind Wandering from Video During Narrative Film Comprehension. In: André, E., Baker, R., Hu, X., Rodrigo, M., du Boulay, B. (eds) Artificial Intelligence in Education. AIED 2017. Lecture Notes in Computer Science(), vol 10331. Springer, Cham. https://doi.org/10.1007/978-3-319-61425-0_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-61425-0_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-61424-3

  • Online ISBN: 978-3-319-61425-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation