Katharina Lingelbach
ABSTRACT
User-aware adaptive systems can greatly benefit from brain-computer interface (BCI) technologies. BCIs allow continuous monitoring of users’ mental states and tailoring of the system to their individual skills and needs. We conducted a feasibility study integrating a BCI using functional near-infrared spectroscopy (fNIRS) into a virtual reality (VR) environment for realistic industrial learning scenarios. Using a fNIRS-based BCI allowed us to a) identify learning progress of individuals based on their working memory load across multiple learning sessions and b) investigate the underlying brain patterns. Our results showed a non-linear relationship between task difficulty and brain responses in the prefrontal cortex (PFC). Finally, we were able to draw four major conclusions regarding architecture components and vital research perspectives, to progress towards a vision of user-aware adaptive system design.
Supplemental Material
- Rumaisa Abu Hasan, Shahida Sulaiman, Nur Nabila Ashykin, Mohd Nasir Abdullah, Yasir Hafeez, and Syed Saad Azhar Ali. 2021. Workplace mental state monitoring during VR-based training for offshore environment. Sensors 21, 14 (2021), 4885.Google Scholar
Cross Ref
- P Aricò, G Borghini, G Di Flumeri, A Colosimo, S Pozzi, and F Babiloni. 2016. A passive brain–computer interface application for the mental workload assessment on professional air traffic controllers during realistic air traffic control tasks. Progress in brain research 228 (2016), 295–328.Google ScholarCross Ref
- Hasan Ayaz, Patricia A Shewokis, Scott Bunce, Kurtulus Izzetoglu, Ben Willems, and Banu Onaral. 2012. Optical brain monitoring for operator training and mental workload assessment. Neuroimage 59, 1 (2012), 36–47.Google ScholarCross Ref
- Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software 67, 1 (2015), 1–48. https://doi.org/10.18637/jss.v067.i01Google ScholarCross Ref
- Benjamin Blankertz, Laura Acqualagna, Sven Dähne, Stefan Haufe, Matthias Schultze-Kraft, Irene Sturm, Marija Ušćumlic, Markus A Wenzel, Gabriel Curio, and Klaus-Robert Müller. 2016. The Berlin brain-computer interface: progress beyond communication and control. Frontiers in neuroscience 10 (2016), 530.Google Scholar
- Matthias Bues, Tobias Schultze, and Benjamin Wingert. 2018. Konzeption und Implementierung einer VR-Lernumgebung für technische Dienstleistungen. In Digitalisierung in der Aus- und Weiterbildung: Virtual und Augmented Reality für Industrie 4.0, Oliver Thomas, Dirk Metzger, and Helmut Niegemann (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 113–123. https://doi.org/10.1007/978-3-662-56551-3_8Google ScholarCross Ref
- Thomas Carlson, Erin Goddard, David M Kaplan, Colin Klein, and J Brendan Ritchie. 2018. Ghosts in machine learning for cognitive neuroscience: Moving from data to theory. NeuroImage 180 (2018), 88–100.Google ScholarCross Ref
- Guy Cheron, Géraldine Petit, Julian Cheron, Axelle Leroy, Anita Cebolla, Carlos Cevallos, Mathieu Petieau, Thomas Hoellinger, David Zarka, Anne-Marie Clarinval, 2016. Brain oscillations in sport: toward EEG biomarkers of performance. Frontiers in psychology 7 (2016), 246.Google Scholar
- Jong-Kwan Choi, Jae-Myoung Kim, Gunpil Hwang, Jaehyeok Yang, Min-Gyu Choi, and Hyeon-Min Bae. 2016. Time-divided spread-spectrum code-based 400 fW-detectable multichannel fNIRS IC for portable functional brain imaging. IEEE Journal of Solid-State Circuits 51, 2 (2016), 484–495.Google ScholarCross Ref
- Caterina Cinel, Davide Valeriani, and Riccardo Poli. 2019. Neurotechnologies for human cognitive augmentation: current state of the art and future prospects. Frontiers in human neuroscience 13 (2019), 13.Google Scholar
- Geoff Cumming and Sue Finch. 2005. Inference by eye: confidence intervals and how to read pictures of data.American psychologist 60, 2 (2005), 170.Google Scholar
- Frédéric Dehais, Alban Duprès, Sarah Blum, Nicolas Drougard, Sébastien Scannella, Raphaëlle N Roy, and Fabien Lotte. 2019. Monitoring pilot’s mental workload using ERPs and spectral power with a six-dry-electrode EEG system in real flight conditions. Sensors 19, 6 (2019), 1324.Google ScholarCross Ref
- Gautier Durantin, J-F Gagnon, Sébastien Tremblay, and Frédéric Dehais. 2014. Using near infrared spectroscopy and heart rate variability to detect mental overload. Behavioural brain research 259 (2014), 16–23.Google Scholar
- Stephen H Fairclough. 2009. Fundamentals of physiological computing. Interacting with computers 21, 1-2 (2009), 133–145.Google ScholarDigital Library
- Marco Ferrari and Valentina Quaresima. 2012. A brief review on the history of human functional near-infrared spectroscopy (fNIRS) development and fields of application. Neuroimage 63, 2 (2012), 921–935.Google ScholarCross Ref
- Frank A Fishburn, Ruth S Ludlum, Chandan J Vaidya, and Andrei V Medvedev. 2019. Temporal derivative distribution repair (TDDR): a motion correction method for fNIRS. Neuroimage 184 (2019), 171–179.Google ScholarCross Ref
- Alexandre Gramfort, Martin Luessi, Eric Larson, Denis A. Engemann, Daniel Strohmeier, Christian Brodbeck, Roman Goj, Mainak Jas, Teon Brooks, Lauri Parkkonen, and Matti S. Hämäläinen. 2013. MEG and EEG Data Analysis with MNE-Python. Frontiers in Neuroscience 7, 267 (2013), 1–13. https://doi.org/10.3389/fnins.2013.00267Google ScholarCross Ref
- Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, Amsterdam, Netherlands, 139–183.Google Scholar
- Stefan Haufe, Jeong-Woo Kim, Il-Hwa Kim, Andreas Sonnleitner, Michael Schrauf, Gabriel Curio, and Benjamin Blankertz. 2014. Electrophysiology-based detection of emergency braking intention in real-world driving. Journal of neural engineering 11, 5 (2014), 056011.Google ScholarCross Ref
- Stefan Haufe, Frank Meinecke, Kai Görgen, Sven Dähne, John-Dylan Haynes, Benjamin Blankertz, and Felix Bießmann. 2014. On the interpretation of weight vectors of linear models in multivariate neuroimaging. Neuroimage 87 (2014), 96–110.Google ScholarCross Ref
- Christian Herff, Dominic Heger, Ole Fortmann, Johannes Hennrich, Felix Putze, and Tanja Schultz. 2014. Mental workload during n-back task—quantified in the prefrontal cortex using fNIRS. Frontiers in human neuroscience 7 (2014), 935.Google Scholar
- Valentin Holzwarth, Johannes Schneider, Joshua Handali, Joy Gisler, Christian Hirt, Andreas Kunz, and Jan vom Brocke. 2021. Towards estimating affective states in virtual reality based on behavioral data. Virtual reality 25, 4 (2021), 1139–1152.Google Scholar
- Alicia Howell-Munson, Emily Doherty, Peter Gavriel, Claire Nicolas, Adam Norton, Rodica Neamtu, Holly Yanco, Yi-Ning Wu, and Erin T Solovey. 2022. Towards Brain Metrics for Improving Multi-Agent Adaptive Human-Robot Collaboration: A Preliminary Study. In 2022 Symposium on Human-Computer Interaction for Work. Association for Computing Machinery, New York, NY, United States, 1–10.Google ScholarDigital Library
- Dongjin Huang, Xianglong Wang, Jinhua Liu, Jinyao Li, and Wen Tang. 2022. Virtual reality safety training using deep EEG-net and physiology data. The visual computer 38, 4 (2022), 1195–1207.Google Scholar
- Nikolaos Kaklanis, Pradipta Biswas, Yehya Mohamad, Mari Feli Gonzalez, Matthias Peissner, Patrick Langdon, Dimitrios Tzovaras, and Christopher Jung. 2016. Towards standardisation of user models for simulation and adaptation purposes. Universal Access in the Information Society 15 (2016), 21–48.Google ScholarDigital Library
- Roy PC Kessels, Martine JE Van Zandvoort, Albert Postma, L Jaap Kappelle, and Edward HF De Haan. 2000. The Corsi block-tapping task: standardization and normative data. Applied neuropsychology 7, 4 (2000), 252–258.Google Scholar
- Jean-Rémi King, Laura Gwilliams, Chris Holdgraf, Jona Sassenhagen, Alexandre Barachant, Denis Alexander Engemann, Eric Larson, and Alexandre Gramfort. 2017. Encoding and Decoding Neuronal Dynamics: Methodological Framework to Uncover the Algorithms of Cognition. HAL open science hal-01848442 (2017), 1–19.Google Scholar
- Marius Klug, Sein Jeung, Anna Wunderlich, Lukas Gehrke, Janna Protzak, Zakaria Djebbara, Andreas Argubi-Wollesen, Bettina Wollesen, and Klaus Gramann. 2022. The BeMoBIL Pipeline for automated analyses of multimodal mobile brain and body imaging data. bioRxiv 1, 1 (2022), 2022–09.Google Scholar
- Christian Kothe, David Medine, Chadwick Boulay, Matthew Grivich, and Tristan Stenner. 2019. Labstreaminglayer. Labstreaminglayer. Retrieved January 04, 2023 from https://labstreaminglayer.readthedocs.io/index.htmlGoogle Scholar
- Wei Lun Lim, Yisi Liu, Salem Chandrasekaran Harihara Subramaniam, Serene Hui Ping Liew, Gopala Krishnan, Olga Sourina, Dimitrios Konovessis, Hock Eng Ang, and Lipo Wang. 2018. EEG-based mental workload and stress monitoring of crew members in maritime virtual simulator. In Transactions on Computational Science XXXII. Springer, Berlin, Heidelberg, 15–28.Google Scholar
- Tania Llana, Cristina Fernandez-Baizan, Magdalena Mendez-Lopez, Camino Fidalgo, and Marta Mendez. 2022. Functional near-infrared spectroscopy in the neuropsychological assessment of spatial memory: A systematic review. Acta Psychologica 224 (2022), 103525.Google ScholarCross Ref
- Fabien Lotte, Josef Faller, Christoph Guger, Yann Renard, Gert Pfurtscheller, Anatole Lécuyer, and Robert Leeb. 2012. Combining BCI with virtual reality: towards new applications and improved BCI. In Towards practical brain-computer interfaces. Springer, Berlin/Heidelberg, Germany, 197–220.Google Scholar
- Robert Luke, Eric D Larson, Maureen J Shader, Hamish Innes-Brown, Lindsey Van Yper, Adrian KC Lee, Paul F Sowman, and David McAlpine. 2021. Analysis methods for measuring passive auditory fNIRS responses generated by a block-design paradigm. Neurophotonics 8, 2 (2021), 025008.Google ScholarCross Ref
- J McAfoose and BT Baune. 2009. Exploring visual–spatial working memory: A critical review of concepts and models. Neuropsychology review 19 (2009), 130–142.Google Scholar
- Ryan McKendrick and Amanda Harwood. 2019. Cognitive workload and workload transitions elicit curvilinear hemodynamics during spatial working memory. Frontiers in Human Neuroscience 13 (2019), 405.Google ScholarCross Ref
- Kim Mens, Rafael Capilla, Nicolás Cardozo, and Bruno Dumas. 2016. A taxonomy of context-aware software variability approaches. In Companion Proceedings of the 15th International Conference on Modularity. Association for Computing Machinery, New York, NY, United States, 119–124.Google ScholarDigital Library
- George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information.Psychological review 63, 2 (1956), 81.Google Scholar
- Guiomar Niso, Elena Romero, Jeremy T Moreau, Alvaro Araujo, and Laurens R Krol. 2023. Wireless EEG: A survey of systems and studies. NeuroImage 269 (2023), 119774.Google ScholarCross Ref
- François Osiurak, Jordan Navarro, and Emanuelle Reynaud. 2018. How our cognition shapes and is shaped by technology: A common framework for understanding human tool-use interactions in the past, present, and future. Frontiers in psychology 9 (2018), 293.Google Scholar
- Evan M Peck, Daniel Afergan, Beste F Yuksel, Francine Lalooses, and Robert JK Jacob. 2014. Using fNIRS to measure mental workload in the real world. In Advances in physiological computing. Springer, Berlin/Heidelberg, Germany, 117–139.Google Scholar
- Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825–2830.Google Scholar
- Stéphanie Philippe, Alexis D Souchet, Petros Lameras, Panagiotis Petridis, Julien Caporal, Gildas Coldeboeuf, and Hadrien Duzan. 2020. Multimodal teaching, learning and training in virtual reality: a review and case study. Virtual Reality & Intelligent Hardware 2, 5 (2020), 421–442.Google ScholarCross Ref
- Maria Piñeyro Salvidegoitia, Nadine Jacobsen, Anna-Katharina R Bauer, Benjamin Griffiths, Simon Hanslmayr, and Stefan Debener. 2019. Out and about: Subsequent memory effect captured in a natural outdoor environment with smartphone EEG. Psychophysiology 56, 5 (2019), e13331.Google ScholarCross Ref
- Paola Pinti, Clarisse Aichelburg, Sam Gilbert, Antonia Hamilton, Joy Hirsch, Paul Burgess, and Ilias Tachtsidis. 2018. A review on the use of wearable functional near-infrared spectroscopy in naturalistic environments. Japanese Psychological Research 60, 4 (2018), 347–373.Google ScholarCross Ref
- Kathrin Pollmann, Oilver Stefani, Amelie Bengsch, Matthias Peissner, and Mathias Vukelić. 2019. How to work in the car of the future? A neuroergonomical study assessing concentration, performance and workload based on subjective, behavioral and neurophysiological insights. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, United States, 1–14.Google Scholar
- Jos Prinsen, Ethel Pruss, Anita Vrins, Caterina Ceccato, and Maryam Alimardani. 2022. A Passive Brain-Computer Interface for Monitoring Engagement during Robot-Assisted Language Learning. In 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, IEEE, Prague, Czech Republic, 1967–1972.Google ScholarCross Ref
- Felix Putze, Christian Herff, Christoph Tremmel, Tanja Schultz, and Dean J Krusienski. 2019. Decoding mental workload in virtual environments: a fNIRS study using an immersive n-back task. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, IEEE, Berlin, Germany, 3103–3106.Google ScholarCross Ref
- Julian E Reiser, Edmund Wascher, Gerhard Rinkenauer, and Stefan Arnau. 2021. Cognitive-motor interference in the wild: Assessing the effects of movement complexity on task switching using mobile EEG. European Journal of Neuroscience 54, 12 (2021), 8175–8195.Google ScholarCross Ref
- Raphaëlle N Roy, Marcel F Hinss, Ludovic Darmet, Simon Ladouce, Emilie S Jahanpour, Bertille Somon, Xiaoqi Xu, Nicolas Drougard, Frédéric Dehais, and Fabien Lotte. 2022. Retrospective on the First Passive Brain-Computer Interface Competition on Cross-Session Workload Estimation. Frontiers in Neuroergonomics 3 (2022), 1–8.Google ScholarCross Ref
- Richard M Ryan and Edward L Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.American psychologist 55, 1 (2000), 68.Google Scholar
- Yangming Shi, Yibo Zhu, Ranjana K Mehta, and Jing Du. 2020. A neurophysiological approach to assess training outcome under stress: A virtual reality experiment of industrial shutdown maintenance using Functional Near-Infrared Spectroscopy (fNIRS). Advanced Engineering Informatics 46 (2020), 101153.Google ScholarCross Ref
- Mojtaba Soltanlou, Maria A Sitnikova, Hans-Christoph Nuerk, and Thomas Dresler. 2018. Applications of functional near-infrared spectroscopy (fNIRS) in studying cognitive development: The case of mathematics and language. Frontiers in psychology 9 (2018), 277.Google Scholar
- Martin Spüler, Tanja Krumpe, Carina Walter, Christian Scharinger, Wolfgang Rosenstiel, and Peter Gerjets. 2017. Brain-computer interfaces for educational applications. Informational environments: Effects of use, effective designs 1 (2017), 177–201.Google Scholar
- Megan Strait and Matthias Scheutz. 2014. What we can and cannot (yet) do with functional near infrared spectroscopy. Frontiers in neuroscience 8 (2014), 117.Google Scholar
- Christoph Tremmel, Christian Herff, Tetsuya Sato, Krzysztof Rechowicz, Yusuke Yamani, and Dean J Krusienski. 2019. Estimating cognitive workload in an interactive virtual reality environment using EEG. Frontiers in human neuroscience 13 (2019), 401.Google Scholar
- Anirudh Unni, Klas Ihme, Meike Jipp, and Jochem W Rieger. 2017. Assessing the driver’s current level of working memory load with high density functional near-infrared spectroscopy: A realistic driving simulator study. Frontiers in human neuroscience 11 (2017), 167.Google Scholar
- Ross E Vanderwert and Charles A Nelson. 2014. The use of near-infrared spectroscopy in the study of typical and atypical development. Neuroimage 85 (2014), 264–271.Google ScholarCross Ref
- Kevin J Verdière, Raphaëlle N Roy, and Frédéric Dehais. 2018. Detecting pilot’s engagement using fNIRS connectivity features in an automated vs. manual landing scenario. Frontiers in human neuroscience 12 (2018), 6.Google Scholar
- Alexander von Lühmann, Antonio Ortega-Martinez, David A Boas, and Meryem Ayşe Yücel. 2020. Using the general linear model to improve performance in fNIRS single trial analysis and classification: a perspective. Frontiers in human neuroscience 14 (2020), 1–17.Google ScholarCross Ref
- Alexander von Lühmann, Yilei Zheng, Antonio Ortega-Martinez, Swathi Kiran, David C Somers, Alice Cronin-Golomb, Louis N Awad, Terry D Ellis, David A Boas, and Meryem A Yücel. 2021. Toward Neuroscience of the Everyday World (NEW) using functional near-infrared spectroscopy. Current opinion in biomedical engineering 18 (2021), 100272.Google Scholar
- Mathias Vukelić. 2021. Connecting Brain and Machine: The Mind Is the Next Frontier. Clinical Neurotechnology meets Artificial Intelligence: Philosophical, Ethical, Legal and Social Implications 1 (2021), 215–226.Google Scholar
- Mathias Vukelić, Katharina Lingelbach, Kathrin Pollmann, and Matthias Peissner. 2020. Oscillatory EEG Signatures of Affective Processes during Interaction with Adaptive Computer Systems. Brain Sciences 11, 1 (2020), 35.Google ScholarCross Ref
- Carina Walter, Wolfgang Rosenstiel, Martin Bogdan, Peter Gerjets, and Martin Spüler. 2017. Online EEG-based workload adaptation of an arithmetic learning environment. Frontiers in human neuroscience 11 (2017), 286.Google Scholar
- Mark S Young, Karel A Brookhuis, Christopher D Wickens, and Peter A Hancock. 2015. State of science: mental workload in ergonomics. Ergonomics 58, 1 (2015), 1–17.Google ScholarCross Ref
- Meryem A Yücel, Alexander v Lühmann, Felix Scholkmann, Judit Gervain, Ippeita Dan, Hasan Ayaz, David Boas, Robert J Cooper, Joseph Culver, Clare E Elwell, 2021. Best practices for fNIRS publications. Neurophotonics 8, 1 (2021), 012101.Google Scholar
- Thorsten O Zander and Christian Kothe. 2011. Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general. Journal of neural engineering 8, 2 (2011), 025005.Google ScholarCross Ref
- Thorsten O Zander, Kunal Shetty, Romy Lorenz, Daniel R Leff, Laurens R Krol, Ara W Darzi, Klaus Gramann, and Guang-Zhong Yang. 2017. Automated task load detection with electroencephalography: towards passive brain–computer interfacing in robotic surgery. Journal of Medical Robotics Research 2, 01 (2017), 1750003.Google ScholarCross Ref
- Rob Zink, Borbála Hunyadi, Sabine Van Huffel, and Maarten De Vos. 2016. Mobile EEG on the bike: disentangling attentional and physical contributions to auditory attention tasks. Journal of neural engineering 13, 4 (2016), 046017.Google ScholarCross Ref
Index Terms
- Towards User-Aware VR Learning Environments: Combining Brain-Computer Interfaces with Virtual Reality for Mental State Decoding
Recommendations
Towards ambulatory brain-computer interfaces: a pilot study with P300 signals
ACE '09: Proceedings of the International Conference on Advances in Computer Entertainment TechnologyBrain-Computer Interfaces (BCI) are communication systems that enable users to interact with computers using only brain activity. This activity is generally measured by ElectroEncephaloGraphy (EEG). A major limitation of BCI is the electrical ...
An SSVEP-Based Brain-Computer Interface to Navigate in a Virtual Home
ICBBB '17: Proceedings of the 7th International Conference on Bioscience, Biochemistry and BioinformaticsIn this paper, an Steady State Visually Evoked Potential (SSVEP)-based Brain-Computer Interface (BCI) is designed for the purpose of navigating in a virtual environment. This BCI system is non-invasive and synchronous. It receives the ...
Feature Analysis of EEG Based Brain-Computer Interfaces to Detect Motor Imagery
Brain InformaticsAbstractBrain-Computer Interfaces (BCI) is one of the alluring breakthroughs for mankind as it provides a new way of communication for the patients of neuro-muscular disorders. Electroencephalography (EEG) signals are the most studied type of signals to ...
Comments