Skip to main content
SpringerLink
Log in

Virtual reality interaction based on visual attention and kinesthetic information

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

Emerging multimedia technologies significantly enhance the naturalness and immersion of human computer interaction. Currently, research on kinesthetic information has gained increasing attentions of multimedia community. However, effective interaction between kinesthetic and other multimedia signals remains a challenging task. In this paper, we propose a visual-kinesthetic interaction in Virtual Reality (VR) and real-world control tasks. First, we model the correlation between user’s visual attention and kinesthetic positions under different tasks. Second, we utilize an attention-based Long Short-Term Memory network to predict the kinesthetic positions. Third, we build a VR system with robotic car control, which validates our model in VR interaction and control tasks. With a high task achievement rate, we envision the implementation of kinesthetic information in a more natural interaction system. The VR interaction system based on the proposed model can also provide guidance for the design of immersive robot teleoperation systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Bhardwaj A, Cizmeci B, Steinbach E, Liu Q et al. (2017) A candidate hardware and software reference setup for kinesthetic codec standardization. In: 2017 IEEE international symposium on haptic, audio and visual environments and games (HAVE), 2017, pp: 1–6.

  • Biswas P, Langdon PM (2014) Multimodal target prediction. In: Stephanidis C, Antona M. (eds) Universal access in human-computer interaction. Design and development methods for universal access. UAHCI 2014. Lecture notes in computer science 8513. Springer, Cham.

  • Chen D, Song A, Tian L, Yu Y, Zhu L (2018) MH-Pen: a pen-type multi-mode haptic interface for touch screens interaction. IEEE Trans Haptics 11(4):555–567

    Google Scholar 

  • Culbertson H, Schorr SB, Okamura AM (2018) Haptics: the present and future of artificial touch sensation. Annual Rev Control, Robotics, Auton Syst 1:385–406

    Article  Google Scholar 

  • Van Damme S, Vega MT, De Turck F (2020) Human-centric quality management of immersive multimedia applications. In: 2020 6th IEEE conference on network softwarization (NetSoft), Ghent, Belgium, 2020, pp 57–64.

  • Falco P, Lu S, Natale C, Pirozzi S, Lee D (2019) A transfer learning approach to cross-modal object recognition: from visual observation to robotic haptic exploration. IEEE Trans Rob 35(4):987–998

    Article  Google Scholar 

  • Gao L, Guo Z, Zhang H, Xu X, Shen HT (2017) Video captioning with attention-based LSTM and semantic consistency. IEEE Trans Multimedia 19(9):2045–2055

    Article  Google Scholar 

  • Gokhale V, Nair J, Chaudhuri S (2016) Opportunistic adaptive haptic sampling on forward channel in telehaptic communication. In: 2016 IEEE haptics symposium (HAPTICS), Philadelphia, PA, USA, 2016, pp. 217–222.

  • Hansen DW, Ji Q (2010) In the eye of the beholder: a survey of models for eyes and gaze. IEEE Trans Pattern Anal Mach Intell 32(3):478–500

    Article  Google Scholar 

  • Huang H, Wang X, Hu M, Tao Y (2019) Applied to mobile multimedia intelligent speech system interactive topic guiding model. IEEE Access 7:182348–182356

    Article  Google Scholar 

  • Isnard V, Nguyen T, Viaud-Delmon I (2021) Exploiting voice transformation to promote interaction in virtual environments. In: 2021 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops (VRW), Lisbon, Portugal, 2021, pp. 75–79.

  • Luong T, Pham H, DManning C (2015) Effective approaches to attention-based neural machine translation. In: 2015 Conference on empirical methods in natural language processing (EMNLP), 2015, pp:1412–1421.

  • Mao L, Zhu P (2017) The medical service robot interaction based on kinect. In: 2017 IEEE International conference on intelligent techniques in control, optimization and signal processing (INCOS), Srivilliputtur, India, 2017, pp. 1–7.

  • Martinez J, Griffiths D, Biscione V, Georgiou O, Carter T (2018). Touchless haptic feedback for supernatural VR experiences. In: 2018 IEEE conference on virtual reality and 3D user interfaces (VR), Tuebingen/Reutlingen, Germany, 2018, pp. 629–630.

  • Pai YS, Dingler T, Kunze K (2019) Assessing hands-free interactions for VR using eye gaze and electromyography. Virtual Reality 23:119–131

    Article  Google Scholar 

  • Pietra AL, Rull MV, Etzi R et al (2021) Promoting eco-driving behavior through multisensory stimulation: a preliminary study on the use of visual and haptic feedback in a virtual reality driving simulator. Virtual Reality 25:945–959

    Article  Google Scholar 

  • Thomas L, Nikola B, Tatiana G, Michael K, Andre B, Anke H, Andreas N (2017) Towards identifying user intentions in exploratory search using gaze and pupil tracking. In: 2017 Conference on conference human information interaction and retrieval ,Oslo Norway, 2017, pp. 273–276.

  • Walker ME, Hedayati H, Szafir D (2019) Robot teleoperation with augmented reality virtual surrogates. In: 14th ACM/IEEE international conference on human-robot interaction (HRI), Daegu, Korea (South), 2019, pp. 202–210.

  • Wang D, Ohnishi K, Xu W (2020) Multimodal haptic display for virtual reality: a survey. IEEE Trans Industr Electron 67(1):610–623

    Article  Google Scholar 

  • Xu X, Song A, Ni D, Lin H, Xiong P, Zhu C (2016) Visual-haptic aid teleoperation based on 3-d environment modeling and updating. IEEE Trans Industr Electron 63(10):6419–6482

    Article  Google Scholar 

  • Xue H, Zhao T, Chen W, Liu Q, Zheng S, Chen CW (2019) Visual attention and haptic control: a cross-study. In: 2019 IEEE fifth international conference on multimedia big data (BigMM), Singapore, 2019, pp. 111–117.

  • Yang C, Luo J, Liu C, Li M, Dai S (2019) Haptics electromyography perception and learning enhanced intelligence for teleoperated robot. IEEE Trans Autom Sci Eng 16(4):1512–1521

    Article  Google Scholar 

  • Yip HM, Navarro-Alarcon D, Liu Y (2016) Development of an eye-gaze controlled interface for surgical manipulators using eye-tracking glasses. In: IEEE international conference on robotics and biomimetics (ROBIO), Qingdao, China, 2016, pp. 1900–1905.

  • Yu X, He W, Li Q, Li Y, Li B (2021) Human-robot co-carrying using visual and force sensing. IEEE Trans Industr Electron 68(9):8657–8666

    Article  Google Scholar 

  • Yuan Z, Bi T, Muntean G, Ghinea G (2015) Perceived synchronization of mulsemedia services. IEEE Trans Multimedia 17(7):957–966

    Article  Google Scholar 

  • Zinchenko K, Wu CY, Song KT (2017) A study on speech recognition control for a surgical robot. IEEE Trans Industr Inf 13(2):607–615

    Article  Google Scholar 

Download references

Funding

This work was supported in part by Natural Science Foundation of Fujian Province, China (Grants No. 2022J02015).

Author information

Authors and Affiliations

  1. Fujian Key Lab for Intelligent Processing and Wireless Transmission of Media Information, Fuzhou University, Fuzhou, 350108, Fujian, China

    Ying Fang, Yiwen Xu, Yanmin Guo & Tiesong Zhao

  2. Peng Cheng Laboratory, Shenzhen, 518038, Guangdong, China

    Tiesong Zhao

  3. Department of Computer Science and Technology, Dalian University of Technology, Dalian, 116081, Liaoning, China

    Qian Liu

Authors
  1. Ying Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qian Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yiwen Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yanmin Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tiesong Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tiesong Zhao.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Human and animal participants

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Ethics approval

I certify that this manuscript is original. The manuscript has not been published and will not be submitted elsewhere for publication while being considered by Virtual Reality. The study is not split up into several parts to increase the quantity of submissions and submitted to various journals or to one journal over time. Results are presented clearly, honestly, and without fabrication, falsification or inappropriate data manipulation (including image based manipulation). We adhere to discipline-specific rules for acquiring, selecting and processing data. No data, text, or theories by others are presented as if they were our own. We make sure we have permissions for the use of software in our study. The submission has been received explicitly from all co-authors. Authors whose names appear on the submission have contributed sufficiently to the scientific work and therefore share collective responsibility and accountability for the results.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (MP4 3466 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fang, Y., Liu, Q., Xu, Y. et al. Virtual reality interaction based on visual attention and kinesthetic information. Virtual Reality 27, 2183–2193 (2023). https://doi.org/10.1007/s10055-023-00801-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-023-00801-3

Keywords

Search

Navigation