Skip to main content
SpringerLink
Log in

fNIRS-based classification of mind-wandering with personalized window selection for multimodal learning interfaces

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Automatic detection of an individual’s mind-wandering state has implications for designing and evaluating engaging and effective learning interfaces. While it is difficult to differentiate whether an individual is mind-wandering or focusing on the task only based on externally observable behavior, brain-based sensing offers unique insights to internal states. To explore the feasibility, we conducted a study using functional near-infrared spectroscopy (fNIRS) and investigated machine learning classifiers to detect mind-wandering episodes based on fNIRS data, both on an individual level and a group level, specifically focusing on automated window selection to improve classification results. For individual-level classification, by using a moving window method combined with a linear discriminant classifier, we found the best windows for classification and achieved a mean F1-score of 74.8%. For group-level classification, we proposed an individual-based time window selection (ITWS) algorithm to incorporate individual differences in window selection. The algorithm first finds the best window for each individual by using embedded individual-level classifiers and then uses these windows from all participants to build the final classifier. The performance of the ITWS algorithm is evaluated when used with eXtreme gradient boosting, convolutional neural networks, and deep neural networks. Our results show that the proposed algorithm achieved significant improvement compared to the previous state of the art in terms of brain-based classification of mind-wandering, with an average F1-score of 73.2%. This builds a foundation for mind-wandering detection for both the evaluation of multimodal learning interfaces and for future attention-aware systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Friedman JH (1989) Regularized discriminant analysis. J Am Stat Assoc 84(405):165–175

  2. Afergan D, Peck EM, Solovey ET, Jenkins A, Hincks SW, Brown ET, Chang R, Jacob RJ (2014) Dynamic difficulty using brain metrics of workload. In: Proceedings of the 32nd annual ACM conference on Human factors in computing systems—CHI’14, pp 3797–3806. https://doi.org/10.1145/2556288.2557230

  3. Balakrishnama S, Ganapathiraju A (1998) Linear discriminant analysis—a brief tutorial. Inst Signal Inf Process 18(4):1–8

    Google Scholar 

  4. Bandara D, Velipasalar S, Bratt S, Hirshfield L (2018) Building predictive models of emotion with functional near-infrared spectroscopy. Int J Hum–Comput Stud 110:75–85

    Article  Google Scholar 

  5. Bixler R, DMello S (2016) Automatic gaze-based user-independent detection of mind wandering during computerized reading. User Model User-Adapt Interact 26(1):33–68. https://doi.org/10.1007/s11257-015-9167-1

    Article  Google Scholar 

  6. Blanchard N, Bixler R, Joyce T, D’Mello S (2014) Automated physiological-based detection of mind wandering during learning. In: International conference on intelligent tutoring systems. Springer, Cham, pp 55–60. https://doi.org/10.1007/978-3-319-07221-0_7

  7. Bosch N, Dmello S (2019) Automatic detection of mind wandering from video in the lab and in the classroom. IEEE Trans Affect Comput. https://doi.org/10.1109/taffc.2019.2908837

    Article  Google Scholar 

  8. Buccino AP, Keles HO, Omurtag A (2016) Hybrid EEG-fNIRS asynchronous brain–computer interface for multiple motor tasks. PLoS ONE 11(1):1–16. https://doi.org/10.1371/journal.pone.0146610

    Article  Google Scholar 

  9. Champaign J, McCalla G (2015) AttentiveLearner: improving mobile MOOC learning via implicit heart rate tracking. In: International conference on artificial intelligence in education. Springer, Cham, pp 367–376. https://doi.org/10.1007/978-3-319-19773-9

  10. Chance B, Anday E, Nioka S, Zhou S, Hong L, Worden K, Li C, Murray T, Ovetsky Y, Pidikiti D, Thomas R (1998) A novel method for fast imaging of brain function, non-invasively, with light. Opt Express 2(10):411. https://doi.org/10.1364/oe.2.000411

    Article  Google Scholar 

  11. Chawla Keven NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: Synthetic Minority Over-sampling Technique Nitesh. J Artif Intell Res 16(1):321–357. https://doi.org/10.1613/jair.953

    Article  MATH  Google Scholar 

  12. Chen T, Guestrin C (2016) XGBoost: a scalable tree boosting system. In: Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining 13–17 August, pp 785–794. https://doi.org/10.1145/2939672.2939785

  13. Cho BH, Lee JM, Ku J, Jang DP, Kim J, Kim IY, Lee JH, Kim SI (2002) Attention enhancement system using virtual reality and EEG biofeedback. In: Proceedings IEEE virtual reality. IEEE, pp 156–163

  14. Christoff K, Gordon AM, Smallwood J, Smith R, Schooler JW (2009) Experience sampling during fMRI reveals default network and executive system contributions to mind wandering. In: Proceedings of the National Academy of Sciences, pp 8719–8724. papers3://publication/uuid/F7FC47FD-5AB1-4FCE-8F30-A99EE1870E01

  15. Combrisson E, Jerbi K (2015) Exceeding chance level by chance: the caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. J Neurosci Methods 250:126–136

    Article  Google Scholar 

  16. Connolly TM, Boyle EA, MacArthur E, Hainey T, Boyle JM (2012) A systematic literature review of empirical evidence on computer games and serious games. Comput Educ 59(2):661–686

    Article  Google Scholar 

  17. Cui X, Bray S, Bryant DM, Glover GH, Reiss AL (2011) A quantitative comparison of nirs and fmri across multiple cognitive tasks. Neuroimage 54(4):2808–2821

    Article  Google Scholar 

  18. D’Mello S, Olney A, Williams C, Hays P (2012) Gaze tutor: a gaze-reactive intelligent tutoring system. Int J Hum–Comput Stud 70(5):377–398

    Article  Google Scholar 

  19. Durantin G, Dehais F, Delorme A (2015) Characterization of mind wandering using fNIRS. Front Syst Neurosci 9:45

    Article  Google Scholar 

  20. Fox KC, Spreng RN, Ellamil M, Andrews-Hanna JR, Christoff K (2015) The wandering brain: meta-analysis of functional neuroimaging studies of mind-wandering and related spontaneous thought processes. NeuroImage 111:611–621. https://doi.org/10.1016/j.neuroimage.2015.02.039

    Article  Google Scholar 

  21. Franklin MS, Smallwood J, Schooler JW (2011) Catching the mind in flight: using behavioral indices to detect mindless reading in real time. Psychon Bull Rev 18(5):992–997. https://doi.org/10.3758/s13423-011-0109-6

    Article  Google Scholar 

  22. Guo Y, Liu Y, Oerlemans A, Lao S, Wu S, Lew MS (2016) Deep learning for visual understanding: a review. Neurocomputing 187:27–48

    Article  Google Scholar 

  23. Harrivel AR, Stephens CL, Milletich RJ, Heinich CM, Last MC, Napoli NJ, Abraham NA, Prinzel LJ, Motter MA, Pope AT (2017) Prediction of cognitive states during flight simulation using multimodal psychophysiological sensing. AIAA Inf Syst AIAA Infotech Aerosp 2017:1–10. https://doi.org/10.2514/6.2017-1135

    Article  Google Scholar 

  24. Herff C, Heger D, Fortmann O, Hennrich J, Putze F, Schultz T (2014) Mental workload during n-back task quantified in the prefrontal cortex using fNIRS. Front Hum Neurosci 7:935

    Article  Google Scholar 

  25. Hirshfield LM, Solovey ET, Girouard A, Kebinger J, Jacob RJ, Sassaroli A, Fantini S (2009) Brain measurement for usability testing and adaptive interfaces: an example of uncovering syntactic workload with functional near infrared spectroscopy. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 2185–2194

  26. Ho TKK, Gwak J, Park CM, Song JI (2019) Discrimination of mental workload levels from multi-channel fNIRS using deep leaning-based approaches. IEEE Access 7:24392–24403. https://doi.org/10.1109/ACCESS.2019.2900127

    Article  Google Scholar 

  27. Hosseini R, Walsh B, Tian F, Wang S (2018) An fNIRS-based feature learning and classification framework to distinguish hemodynamic patterns in children who stutter. IEEE Trans Neural Syst Rehabil Eng 26(6):1254–1263. https://doi.org/10.1109/TNSRE.2018.2829083

    Article  Google Scholar 

  28. Huppert TJ, Diamond SG, Franceschini MA, Boas DA (2009) HomER: a review of time-series analysis methods for near-infrared spectroscopy of the brain. Appl Opt 48(10):33. https://doi.org/10.1364/AO.48.00D280

    Article  Google Scholar 

  29. Hutt S, Mills C, Bosch N, Krasich K, Brockmole J, D’mello S (2017) Out of the Fr-“Eye”-ing Pan: towards gaze-based models of attention during learning with technology in the classroom. In: Proceedings of the 25th conference on user modeling, adaptation and personalization. ACM, pp 94–103. https://doi.org/10.1145/3079628.3079669

  30. Izzetoglu M, Izzetoglu K, Bunce S, Ayaz H, Devaraj A, Onaral B, Pourrezaei K (2005) Functional near-infrared neuroimaging. IEEE Trans Neural Syst Rehabil Eng 13:153–159. https://doi.org/10.1109/TNSRE.2005.847377

    Article  Google Scholar 

  31. Jha AP, Morrison AB, Dainer-Best J, Parker S, Rostrup N, Stanley EA (2015) Minds at attention: mindfulness training curbs attentional lapses in military cohorts. PloS ONE 10(2)

  32. Jin CY, Borst JP, Vugt MKV (2019) Predicting task-general mind-wandering with EEG. Cogn Affect Behav Neurosci 19:1–15

    Article  Google Scholar 

  33. Kawashima I, Kumano H (2017) Prediction of mind-wandering with electroencephalogram and non-linear regression modeling. Front Hum Neurosci 11(July):1–10. https://doi.org/10.3389/fnhum.2017.00365

    Article  Google Scholar 

  34. Kerous B, Skola F, Liarokapis F (2018) Eeg-based bci and video games: a progress report. Virtual Real 22(2):119–135

    Article  Google Scholar 

  35. Khan MJ, Liu X, Bhutta MR, Hong KS (2016) Drowsiness detection using fNIRS in different time windows for a passive BCI. In: 2016 6th IEEE international conference on biomedical robotics and biomechatronics (BioRob). IEEE, pp 227–231. https://doi.org/10.1109/BIOROB.2016.7523628

  36. Killingsworth MA, Gilbert DT (2010) A wandering mind is an unhappy mind. Science 330(6006):932. https://doi.org/10.1126/science.1192439

    Article  Google Scholar 

  37. Makantasis K, Doulamis A, Doulamis N, Nikitakis A, Voulodimos A (2018) Tensor-based nonlinear classifier for high-order data analysis. In: 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 2221–2225

  38. Manly T, Robertson IH, Galloway M, Hawkins K (1999) The absent mind: further investigations of sustained attention to response. Neuropsychologia 37(6):661–670

    Article  Google Scholar 

  39. Mayseless N, Hawthorne G, Reiss AL (2019) Real-life creative problem solving in teams: fNIRS based hyperscanning study. NeuroImage 203(August):116161. https://doi.org/10.1016/j.neuroimage.2019.116161

    Article  Google Scholar 

  40. McKendrick R, Parasuraman R, Ayaz H (2015) Wearable functional near infrared spectroscopy (fNIRS) and transcranial direct current stimulation (tDCS): expanding vistas for neurocognitive augmentation. Front Syst Neurosci. https://doi.org/10.3389/fnsys.2015.00027

    Article  Google Scholar 

  41. McKendrick R, Parasuraman R, Murtza R, Formwalt A, Baccus W, Paczynski M, Ayaz H (2016) Into the wild: neuroergonomic differentiation of hand-held and augmented reality wearable displays during outdoor navigation with functional near infrared spectroscopy. Front Hum Neurosci 10(MAY2016):216. https://doi.org/10.3389/fnhum.2016.00216

    Article  Google Scholar 

  42. Mills C, Mello SD (2015) Toward a real-time (Day) Dreamcatcher: sensor-free detection of mind wandering during online reading. In: International educational data mining society

  43. Mooneyham BW, Schooler JW (2013) The costs and benefits of mind-wandering: a review. Can J Exp Psychol 67(1):11–18. https://doi.org/10.1037/a0031569

    Article  Google Scholar 

  44. Naseer N, Hong KS (2013) Classification of functional near-infrared spectroscopy signals corresponding to the right- and left-wrist motor imagery for development of a brain–computer interface. Neurosci Lett 553:84–89. https://doi.org/10.1016/j.neulet.2013.08.021

    Article  Google Scholar 

  45. Naseer N, Hong KS (2015) fNIRS-based brain–computer interfaces: a review. Front Hum Neurosci 9:3

    Google Scholar 

  46. Naseer N, Hong KS (2015) fNIRS-based brain–computer interfaces: a review. Front Hum Neurosci 9:1–15. https://doi.org/10.3389/fnhum.2015.00172

    Article  Google Scholar 

  47. Nelson BC (2007) Exploring the use of individualized, reflective guidance in an educational multi-user virtual environment. J Sci Educ Technol 16(1):83–97

    Article  Google Scholar 

  48. Ninaus M, Kober SE, Friedrich EV, Dunwell I, Freitas SD, Arnab S, Ott M, Kravcik M, Lim T, Louchart SJJ et al (2014) Neurophysiological methods for monitoring brain activity in serious games and virtual environments: a review. Int J Technol Enhanc Learn 6(1):78

    Article  Google Scholar 

  49. Noori FM, Naseer N, Qureshi NK, Nazeer H, Khan RA (2017) Optimal feature selection from fNIRS signals using genetic algorithms for BCI. Neurosci Lett 647:61–66. https://doi.org/10.1016/j.neulet.2017.03.013

    Article  Google Scholar 

  50. Orihuela-Espina F, Leff DR, James DR, Darzi AW, Yang GZ (2010) Quality control and assurance in functional near infrared spectroscopy (fNIRS) experimentation. Phys Med Biol 55(13):3701–3724. https://doi.org/10.1088/0031-9155/55/13/009

    Article  Google Scholar 

  51. Peck EM, Carlin E, Jacob R (2015) Designing brain–computer interfaces for attention-aware systems. Computer 48(10):34–42. https://doi.org/10.1109/MC.2015.315

    Article  Google Scholar 

  52. Pike MF, Maior HA, Porcheron M, Sharples SC, Wilson ML (2014) Measuring the effect of think aloud protocols on workload using fNIRS. In: Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, pp 3807–3816. https://doi.org/10.1145/2556288.2556974

  53. Pinti P, Scholkmann F, Hamilton A, Burgess P, Tachtsidis I (2018) Current status and issues regarding pre-processing of fNIRS neuroimaging data: an investigation of diverse signal filtering methods within a general linear model framework. Front Hum Neurosci 12:505

    Article  Google Scholar 

  54. Rapp DN (2006) The value of attention aware systems in educational settings. Comput Hum Behav 22(4):603–614

    Article  Google Scholar 

  55. Schmorrow DD, Fidopiastis CM (2015) Phylter: a system for modulating notifications in wearables using physiological sensing. In: International conference on augmented cognition, vol 9183. Springer, Cham, pp 167–177. https://doi.org/10.1007/978-3-319-20816-9

  56. Schooler JW, Smallwood J, Christoff K, Handy TC, Reichle ED, Sayette MA (2011) Meta-awareness, perceptual decoupling and the wandering mind. Trends Cogn Sci 15(7):319–326. https://doi.org/10.1016/j.tics.2011.05.006

    Article  Google Scholar 

  57. Shin J, Müller KR, Hwang HJ (2016) Near-infrared spectroscopy (NIRS)-based eyes-closed brain–computer interface (BCI) using prefrontal cortex activation due to mental arithmetic. Sci Rep 6(October):1–11. https://doi.org/10.1038/srep36203

    Article  Google Scholar 

  58. Shin J, Von Luhmann A, Blankertz B, Kim DW, Jeong J, Hwang HJ, Muller KR (2017) Open access dataset for EEG + NIRS single-trial classification. IEEE Trans Neural Syst Rehabil Eng 25(10):1735–1745. https://doi.org/10.1109/TNSRE.2016.2628057

    Article  Google Scholar 

  59. Shin J, Von Lühmann A, Kim DW, Mehnert J, Hwang HJ, Müller KR (2018) Simultaneous acquisition of eeg and nirs during cognitive tasks for an open access dataset. Sci Data 5:180003

    Article  Google Scholar 

  60. Smallwood J, Schooler JW (2006) The restless mind. Psychol Bull 132(6):946–958. https://doi.org/10.1037/0033-2909.132.6.946

    Article  Google Scholar 

  61. Smallwood J, Fishman DJ, Schooler JW (2007) Counting the cost of an absent mind: mind wandering as an underrecognized influence on educational performance. Psychon Bull Rev 14(2):230–236. https://doi.org/10.3758/BF03194057

    Article  Google Scholar 

  62. Smallwood J, Beach E, Schooler JW, Handy TC (2008) Going awol in the brain: mind wandering reduces cortical analysis of external events. J Cogn Neurosci 20(3):458–469

    Article  Google Scholar 

  63. Solovey ET, Girouard A, Chauncey K, Hirshfield LM, Sassaroli A, Zheng F, Fantini S, Jacob RJK (2009) Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines. In: Proceedings of the 22nd annual ACM symposium on user interface software and technology. ACM

  64. Treacy Solovey E, Afergan D, Peck EM, Hincks SW, Jacob RJ (2015) Designing implicit interfaces for physiological computing: guidelines and lessons learned using fNIRS. ACM Trans Comput–Hum Interact 21(6):35

    Article  Google Scholar 

  65. Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E (2018) Deep learning for computer vision: a brief review. Comput Intell Neurosci 2018:7068349. https://doi.org/10.1155/2018/7068349

  66. Wouters P, Van der Spek ED, Van Oostendorp H (2009) Current practices in serious game research: a review from a learning outcomes perspective. In: Games-based learning advancements for multi-sensory human computer interfaces: techniques and effective practices. IGI Global, pp 232–250

Download references

Author information

Authors and Affiliations

  1. Worcester Polytechnic Institute, Worcester, MA, 01609, USA

    Ruixue Liu & Erin T. Solovey

  2. University of Pittsburgh, Pittsburgh, PA, 15260, USA

    Erin Walker

  3. Drexel University, Philadelphia, PA, 19104, USA

    Leah Friedman

  4. Lehigh University, Bethlehem, PA, 18015, USA

    Catherine M. Arrington

Authors
  1. Ruixue Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Erin Walker
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leah Friedman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Catherine M. Arrington
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Erin T. Solovey
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Erin T. Solovey.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported in part by the U.S. National Science Foundation under Grants NCS-1835307, CNS-1711773 and NCS-1835251.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, R., Walker, E., Friedman, L. et al. fNIRS-based classification of mind-wandering with personalized window selection for multimodal learning interfaces. J Multimodal User Interfaces 15, 257–272 (2021). https://doi.org/10.1007/s12193-020-00325-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-020-00325-z

Keywords

Search

Navigation