skip to main content
10.1145/2207676.2208372acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Brainput: enhancing interactive systems with streaming fnirs brain input

Published: 05 May 2012 Publication History

Abstract

This paper describes the Brainput system, which learns to identify brain activity patterns occurring during multitasking. It provides a continuous, supplemental input stream to an interactive human-robot system, which uses this information to modify its behavior to better support multitasking. This paper demonstrates that we can use non-invasive methods to detect signals coming from the brain that users naturally and effortlessly generate while using a computer system. If used with care, this additional information can lead to systems that respond appropriately to changes in the user's state. Our experimental study shows that Brainput significantly improves several performance metrics, as well as the subjective NASA-Task Load Index scores in a dual-task human-robot activity.

References

[1]
Barbosa, A.O.G., Achanccaray, D.R. and Meggiolaro, M.A., Activation of a mobile robot through a brain computer interface. In Proc. IEEE ICRA, (2010), 48154821.
[2]
Blankertz, B., et al. The Berlin brain-computer interface: Machine-learning based detection of user specific brain states. In G. Dornhege, J. d. R. Millán, T. Hinterberger, D. J. McFarland & K. Müller (Eds.), Toward BrainComputer Interfacing: MIT Press (2007).
[3]
Coyle, S.M., Ward, T.E. and Markham, C.M. Braincomputer interface using a simplified functional nearinfrared spectroscopy system. J. of Neural Engineering,(3). 219.
[4]
Escolano, C., et al. A telepresence robotic system operated with a P300-based brain-computer interface: Initial tests with ALS patients. In Proc. IEEE EMBC, (2010), 4476--4480.
[5]
Fairclough, S.H. Fundamentals of physiological computing. Interact. Comput., 21 (1--2). 133--145.
[6]
Girouard, A., Solovey, E.T. and Jacob, J.K. Designing a Passive Brain Computer Interface using Real Time Classification of Functional Near-Infrared Spectroscopy. Intl J of Autonomous and Adaptive Communications Systems (2010).
[7]
Grimes, D., Tan, D.S., Hudson, S.E., Shenoy, P. and Rao, R.P.N. Feasibility and pragmatics of classifying working memory load with an electroencephalograph. In Proc. CHI'08, ACM Press (2008).
[8]
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P. and Witten, I.H. The WEKA data mining software: an update. SIGKDD Explor. Newsl., 11 (1). 10--18.
[9]
Hart, S.G. and Staveland, L.E. Development of NASATLX (Task Load Index): Results of empirical and theorical research. in Hancock, P., Meshkati, N. ed. Human Mental Workload, 1988, pp 139 - 183.
[10]
Hirshfield, L.M., Solovey, E.T., Girouard, A., Kebinger, J., Sassaroli, A., Tong, Y., Fantini, S. and Jacob, R.J.K. Brain Measurement for Usability Testing and Adaptive Interfaces: An Example of Uncovering Syntactic Workload with Functional Near Infrared Spectroscopy. In Proc. CHI'09, ACM Press (2009).
[11]
Hirshfield, L.M., et al. This is your brain on interfaces: enhancing usability testing with functional near-infrared spectroscopy. In Proc. CHI'11, ACM Press (2011).
[12]
Jackson, M.M. and Mappus, R. Applications for BrainComputer Interfaces. in Tan, D. and Nijholt, A. eds. Brain Computer Interfaces: Applying our Minds to Human-Computer Interaction, Springer (2010), 89--104.
[13]
Kennedy, P.R., Bakay, R.A.E., Moore, M.M., Adams, K. and Goldwaithe, J. Direct control of a computer from the human central nervous system. IEEE Trans. on Rehabilitation Engineering, 8 (2). 198.
[14]
Koechlin, E., Basso, G., Pietrini, P., Panzer, S. & Grafman, J. The role of the anterior prefrontal cortex in human cognition. Nature (1999), 148--151.
[15]
Kuikkaniemi, K., et al. The influence of implicit and explicit biofeedback in first-person shooter games. In Proc CHI'10. ACM Press (2010).
[16]
Lee, J.C. and Tan, D.S. Using a low-cost electroencephalograph for task classification in HCI research. In Proc. UIST, ACM Press (2006).
[17]
Mandryk, R., Atkins, M. and Inkpen, K. A continuous and objective evaluation of emotional experience with interactive play environments. In Proc. CHI, ACM Press (2006).
[18]
Nacke, L.E., Kalyn, M., Lough, C. and Mandryk, R.L. Biofeedback game design: using direct and indirect physiological control to enhance game interaction. In Proc. CH1, ACM Press (2011).
[19]
O'Hara, K., Sellen, A. and Harper, R. Embodiment in brain-computer interaction. In Proc. CHI, ACM Press (2011).
[20]
Okumura, M. and Zhiwei, L., On NIRS-based brainrobot interface. In Proc. IEEE ROBIO (2007), 864--869.
[21]
Pan, M.K.X.J., Chang, J.-S., Himmetoglu, G.H., Moon, A., Hazelton, T.W., MacLean, K.E. and Croft, E.A. Now where was I?: physiologically-triggered bookmarking. In Proc. CHI, ACM Press (2011).
[22]
Perrin, X., Chavarriaga, R., Colas, F., Siegwart, R. and Millán, J.d.R. Brain-coupled interaction for semiautonomous navigation of an assistive robot. Robot. Auton. Syst., 58 (12). 1246--1255.
[23]
Pfurtscheller, G., Flotzinger, D. and Kalcher, J. Braincomputer interface: a new communication device for handicapped persons. J. Microcomput. Appl., 16 (3). 293--299.
[24]
Pope, A. T., et al. Biocybernetic system evaluates indices of operator engagement in automated task. Biological Psychology, 40, (1995), 187--195.
[25]
Schalk, G., McFarland, D.J., Hinterberger, T., Birbaumer, N. and Wolpaw, J.R. BCI2000: a generalpurpose brain-computer interface (BCI) system. IEEE Trans. on Biomedical Engineering, 51 (6). 1034.
[26]
Schermerhorn, P. and Scheutz, M. Disentangling the Effects of Robot Affect, Embodiment, and Autonomy on Human Team Members in a Mixed-Initiative Task. In Proc. ACHI (2011), 236--241.
[27]
Scheutz, M., Schermerhorn, P. and Kramer, J. The utility of affect expression in natural language interactions in joint human-robot tasks. In Proc. HRI (2006).
[28]
Solovey, E.T., et al. Using fNIRS Brain Sensing in Realistic HCI Settings: Experiments and Guidelines. In Proc. UIST (2009).
[29]
Solovey, E.T., et al., Sensing Cognitive Multitasking for a Brain-Based Adaptive User Interface. In Proc. CHI, ACM Press (2011).
[30]
Solovey, E. T. Real-time fNIRS Brain Input for Enhancing Interactive Systems. Doctoral Dissertation, Tufts University (2012).
[31]
Tsubone, T., Tsutsui, K., Muroga, T. and Wada, Y. Estimation of Force Motor Command to Control Robot by NIRS-Based BCI. in Masumi, I., Kenji, D., Hiroyuki, M. and Takeshi, Y. eds. Neural Information Processing, Springer-Verlag (2008), 969--978.
[32]
Wolpaw, J.R., McFarland, D.J., Neat, G.W. and Forneris, C.A. An EEG-based brain-computer interface for cursor control. Electroencephalography and Clinical Neurophysiology, 78 (3). 252.
[33]
Yuksel, B.F., Donnerer, M., Tompkin, J. and Steed, A. A novel brain-computer interface using a multi-touch surface. In Proc. CHI, ACM Press (2010).

Cited By

View all
  • (2024)NeuroCHI: Are We Prepared for the Integration of the Brain with Computing?Extended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3643973(1-5)Online publication date: 11-May-2024
  • (2024)Crowdsourcing Affective Annotations Via fNIRS-BCIIEEE Transactions on Affective Computing10.1109/TAFFC.2023.327391615:1(297-308)Online publication date: Jan-2024
  • (2023)Brain-Computer Integration: A Framework for the Design of Brain-Computer Interfaces from an Integrations PerspectiveACM Transactions on Computer-Human Interaction10.1145/360362130:6(1-48)Online publication date: 25-Sep-2023
  • Show More Cited By

Index Terms

  1. Brainput: enhancing interactive systems with streaming fnirs brain input

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
    May 2012
    3276 pages
    ISBN:9781450310154
    DOI:10.1145/2207676
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 05 May 2012

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. brain computer interface
    2. fnirs
    3. human-robot interaction
    4. multitasking
    5. near-infrared spectroscopy

    Qualifiers

    • Research-article

    Conference

    CHI '12
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)54
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 20 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)NeuroCHI: Are We Prepared for the Integration of the Brain with Computing?Extended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3643973(1-5)Online publication date: 11-May-2024
    • (2024)Crowdsourcing Affective Annotations Via fNIRS-BCIIEEE Transactions on Affective Computing10.1109/TAFFC.2023.327391615:1(297-308)Online publication date: Jan-2024
    • (2023)Brain-Computer Integration: A Framework for the Design of Brain-Computer Interfaces from an Integrations PerspectiveACM Transactions on Computer-Human Interaction10.1145/360362130:6(1-48)Online publication date: 25-Sep-2023
    • (2023)Joie: a Joy-based Brain-Computer Interface (BCI)Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606761(1-14)Online publication date: 29-Oct-2023
    • (2023)Impact of BCI-Informed Visual Effect Adaptation in a Walking SimulatorProceedings of the 18th International Conference on the Foundations of Digital Games10.1145/3582437.3582448(1-8)Online publication date: 12-Apr-2023
    • (2023)A Survey on Measuring Cognitive Workload in Human-Computer InteractionACM Computing Surveys10.1145/358227255:13s(1-39)Online publication date: 13-Jul-2023
    • (2023)Toward Workload-Based Adaptive Automation: The Utility of fNIRS for Measuring Load in Multiple Resources in the BrainInternational Journal of Human–Computer Interaction10.1080/10447318.2023.226624240:22(7404-7430)Online publication date: 23-Oct-2023
    • (2022)A Real-Time Fatigue Sensing and Enhanced Feedback SystemInformation10.3390/info1305023013:5(230)Online publication date: 30-Apr-2022
    • (2022)Towards Brain Metrics for Improving Multi-Agent Adaptive Human-Robot Collaboration: A Preliminary StudyProceedings of the 1st Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3533406.3533419(1-10)Online publication date: 8-Jun-2022
    • (2022)Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and ReuseACM Transactions on Computer-Human Interaction10.1145/349055429:4(1-43)Online publication date: 31-Mar-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media