Skip to main content
SpringerLink
Log in

An exploratory study of multimodal interaction modeling based on neural computation

基于神经计算的多通道交互模型的探索性研究

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Multimodal interaction serves an important role in human-computer interaction. In this paper we propose a multimodal interaction model based on the latest cognitive research findings. The proposed model combines two proven neural computations, and helps to reveal the enhancement or depression influence of multimodal presentation upon the corresponding interaction task performance. A set of experiments is designed and conducted within the constraints of the model, which demonstrates the observed performance enhancement and depression effects. Our exploration and the experimental results help to further solve the question about how tactile feedback signal contribute the multimodal interaction efficiency which could provide guidelines for designing the tactile feedback in multimodal interaction.

摘要

创新点

多通道交互在人机交互中具有重要的作用。 本文基于最新的认知研究成果, 提出了一种多通道交互模型。 该模型把两个已被证明的神经计算相结合, 用于揭示不同的多通道呈现对相应的交互任务绩效所产生的增强或抑制效果。 本文在该模型的适用约束下设计并实现了一组实验, 实验得出观察到的绩效增强效应和观察到的绩效抑制效应。 本文的探索思路和实验结果有助于进一步解决触觉反馈信息对多通道交互效率的贡献问题, 从而为多通道交互中触觉反馈的设计提供指导。

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Jaimes A, Sebe N. Multimodal human-computer interaction: a survey. Comput Vis Image Und, 2007, 108: 116–134

    Article  Google Scholar 

  2. Li M Z, Dai G Z, Dong S H. Software model and interaction algorithm of multimodal interface (in Chinese). Chinese J Comput, 1998, 21: 111–118

    Google Scholar 

  3. Salvendy G. Handbook of Human Factors and Ergonomics. Hoboken: John Wiley & Sons, 2012

    Book  Google Scholar 

  4. Ware C. Information Visualization: Perception for Design. San Francisco: Morgan Kaufmann Publishers Incorporated, 2013

    Google Scholar 

  5. Shneiderman B, Plaisant C, Cohen M, et al. Designing the User Interface: Strategies for Effective Human-Computer Interaction. 5th ed. Upper Saddle River: Addison-Wesley, 2009

    Google Scholar 

  6. Wang H A, Ma C X. Interactive multi-scale structures for summarizing video content. Sci China Inf Sci, 2013, 56: 052108

    Google Scholar 

  7. Gallace A, Spence C. In Touch With the Future: the Sense of Touch From Cognitive Neuroscience to Virtual Reality. Oxford: Oxford University Press, 2014

    Book  Google Scholar 

  8. Gallace A, Tan Z H, Spence C. The body surface as a communication system: the state of the art after 50 years. Presence, 2007, 16: 655–676

    Article  Google Scholar 

  9. Bau O, Poupyrev I, Israr A, et al. TeslaTouch: electrovibration for touch surfaces. In: Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology. New York: ACM Press, 2010. 283–292

    Chapter  Google Scholar 

  10. McGrath B J, Estrada A, Braithwaite M B, et al. Tactile Situation Awareness System Flight Demonstration. Army Aeromedical Resaerch Lab Fort Rucker AL Technical Report USAARL 2004-10. 2004

    Google Scholar 

  11. Lu L, Tian F, Dai G Z, et al. A study of the multimodal congnition and interaction based on touch, audition and vision (in Chinese). J Comput Aided Design Comput Graph, 2014, 26: 654–661

    Google Scholar 

  12. McGurk H, MacDonald J. Hearing lips and seeing voices. Nature, 1976, 264, 746–748

    Article  Google Scholar 

  13. Mayer R E. Multimedia learning. Psychol Learn Motiv, 2002, 41: 85–139

    Article  Google Scholar 

  14. Kieras D E, Meyer D E. An overview of the EPIC architecture for cognition and performance with application to human-computer interaction. Human-Comput Interact, 1997, 12: 391–438

    Article  Google Scholar 

  15. Ohshiro T, Angelaki D E, de Angelis G C. A normalization model of multisensory integration. Nature Neurosci, 2011, 14: 775–782

    Article  Google Scholar 

  16. Carandini M. From circuits to behavior: a bridge too far? Nature Neurosci, 2012, 15: 507–509

  17. Vroomen J, Keetels M. Perception of intersensory synchrony: a tutorial review. Atte Perce Psycho, 2010, 72: 871–884

    Article  Google Scholar 

  18. Calvert G A, Spence C, Stein B E. The Handbook of Multisensory Processes. Cambridge: MIT Press, 2004

    Google Scholar 

  19. Ernst M O, Bülthoff H H. Merging the senses into a robust percept. Trends Cogn Sci, 2004, 8: 162–169

    Article  Google Scholar 

  20. Stein B E, Stanford T R, Rowland B A. The neural basis of multisensory integration in the midbrain: its organization and maturation. Hearing Res, 2009, 258: 4–15

    Article  Google Scholar 

  21. Carandini M, Heeger D J. Normalization as a canonical neural computation. Nature Rev Neurosci, 2012, 13: 51–62

    Article  Google Scholar 

  22. Stein B E, Stanford T R. Multisensory integration: current issues from the perspective of the single neuron. Nature Rev Neurosci, 2008, 9: 255–266

    Article  Google Scholar 

  23. Molholm S, Ritter W, Murray M M, et al. Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn Brain Res, 2002, 14: 115–128

    Article  Google Scholar 

  24. Baranek G T. Efficacy of sensory and motor interventions for children with autism. J Autism Develop Disord, 2002, 32: 397–422

    Article  Google Scholar 

  25. Brown S, Shankar R, Smith K. Borderline personality disorder and sensory processing impairment. Progress Neurol Psychiat, 2009, 13: 10–16

    Article  Google Scholar 

  26. Stein B E, Meredith M A. The Merging of the Senses. Cambridge: MIT Press, 1993

    Google Scholar 

  27. Meredith M A, Stein B E. Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol, 1996, 75: 1843–1857

    Google Scholar 

  28. Meredith M A, Nemitz J W, Stein B E. Determinants of multisensory integration in superior colliculus neurons. I. temporal factors. J Neurosci, 1987, 7: 3215–3229

    Google Scholar 

  29. Meredith M A, Stein B E. Interactions among converging sensory inputs in the superior colliculus. Science, 1983, 221: 389–391

    Article  Google Scholar 

  30. Spence C, Driver J. Crossmodal Space and Crossmodal Attention. Oxford: Oxford University Press, 2004

    Book  Google Scholar 

  31. Gillmeister H, Eimer M. Tactile enhancement of auditory detection and perceived loudness. Brain Res, 2007, 1160: 58–68

    Article  Google Scholar 

  32. Heeger D J. Normalization of cell responses in cat striate cortex. Visual Neurosci, 1992, 9: 181–197

    Article  Google Scholar 

  33. Körding K P, Wolpert D M. Bayesian integration in sensorimotor learning. Nature, 2004, 427: 244–247

    Article  Google Scholar 

  34. Ernst M O, Banks M S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 2002, 415: 429–433

    Article  Google Scholar 

  35. Sternberg R. Cognitive Psychology. 3rd ed. Carolina: Wadsworth Publishing, 2003

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Beijing Key Laboratory of Human-Computer Interaction, Institute of Software, Chinese Academy of Sciences, Beijing, 100190, China

    Lu Lu, Fei Lyu, Yineng Chen, Guozhong Dai & Hongan Wang

  2. University of Chinese Academy of Sciences, Beijing, 100190, China

    Lu Lu & Yineng Chen

  3. State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing, 100190, China

    Feng Tian & Hongan Wang

Authors
  1. Lu Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fei Lyu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Feng Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yineng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guozhong Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hongan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Feng Tian.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, L., Lyu, F., Tian, F. et al. An exploratory study of multimodal interaction modeling based on neural computation. Sci. China Inf. Sci. 59, 92106 (2016). https://doi.org/10.1007/s11432-016-5520-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-016-5520-1

Keywords

关键词

Search

Navigation