Skip to main content
SpringerLink
Log in

Human–Robot Collaborative Control in a Virtual-Reality-Based Telepresence System

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

In this paper, we develop a robotic telepresence system to provide remote users with immersive embodiment in local environments through a custom-designed mobile robot. The proposed telepresence system uses a virtual reality (VR) device to connect a remote user to the robot. Three dimensional visual data from a RGB-D camera are rendered for real-time stereoscopic display in the VR device, which forms a deeply-coupled human machine system and creates an immersive experience of telepresence. Based on a user study, it is found that better user experience can be achieved by allowing the robot to track the speaker while being aware of the intention of the remote user. To this end we propose a human-robot collaborative control framework based on human intention recognition and sound localization. The intentions of head movement of the remote user are inferred based on the motion of the VR device using hidden Markov models. The speaker is tracked through sound source localization using a microphone array. A collaborative control scheme is developed to fuse the control from the robot and the remote user. Experiments are conducted in both one-to-one and one-to-two remote conversation scenarios. The results show that the proposed system can significantly improve the immersiveness and performance of robotic telepresence systems, therefore greatly enhancing the user experience of such telepresence systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Goodrich MA, Schultz AC (2007) Human-robot interaction: a survey. Found Trends Hum-Comput Interact 1(3):203

    Article  Google Scholar 

  2. Tsui KM, Desai M, Yanco H, Uhlik C et al (2011) Exploring use cases for telepresence robots. In: Human–Robot Interaction (HRI), 2011 6th ACM/IEEE international conference on. IEEE, pp 11–18

  3. Kristoffersson A, Coradeschi S, Loutfi A (2013) A review of mobile robotic telepresence. Adv Hum-Comput Interact 2013:3

    Article  Google Scholar 

  4. Paulos E, Canny J (2001) Social tele-embodiment: understanding presence. Auton Robots 11(1):87

    Article  Google Scholar 

  5. Michaud F, Boissy P, Labonte D, Corriveau H, Grant A, Lauria M, Cloutier R, Roux MA, Iannuzzi D, Royer MP (2007) Telepresence robot for home care assistance. In: AAAI Spring symposium: multidisciplinary collaboration for socially assistive robotics, California, USA, pp 50–55

  6. Google Cardboard (2015). https://www.google.com/get/cardboard/

  7. Oculus Rift (2015). https://www.oculus.com

  8. Adalgeirsson SO, Breazeal C (2010) Mebot: a robotic platform for socially embodied presence. In: Proceedings of the 5th ACM/IEEE international conference on Human–robot interaction. IEEE Press, pp 15–22

  9. Escolano C, Antelis JM, Minguez J (2012) A telepresence mobile robot controlled with a noninvasive brain–computer interface. IEEE Trans Syst Man Cybern Part B Cybern 42(3):793

    Article  Google Scholar 

  10. Sirkin D, Ju W (2012) Consistency in physical and on-screen action improves perceptions of telepresence robots. In: Proceedings of the seventh annual ACM/IEEE international conference on human–robot interaction. ACM, pp 57–64

  11. Martins H, Oakley I, Ventura R (2014) Design and evaluation of a head-mounted display for immersive 3D teleoperation of field robots. Robotica 33:2166

    Article  Google Scholar 

  12. Kratz S, Vaughan J, Mizutani R, Kimber D (2015) Evaluating stereoscopic video with head tracking for immersive teleoperation of mobile telepresence robots. In: Proceedings of the tenth annual ACM/IEEE international conference on human–robot interaction extended abstracts. ACM, pp 43–44

  13. Kato Y (2015) A remote navigation system for a simple tele-presence robot with virtual reality. In: 2015 IEEE/RSJ international conference on Intelligent Robots and Systems (IROS). IEEE, pp 4524–4529

  14. DORA (2020). http://doraplatform.com

  15. Stiefelhagen R, Yang J, Waibel A (2001) Estimating focus of attention based on gaze and sound. In: Proceedings of the 2001 workshop on Perceptive user interfaces, ACM, pp 1–9

  16. Gao M, Oberlander J, Schamm T, Zollner JM (2014) Contextual task-aware shared autonomy for assistive mobile robot teleoperation. In: 2014 IEEE/RSJ international conference on Intelligent Robots and Systems (IROS 2014). IEEE , pp 3311–3318

  17. Carlson T, Monnard G, del R. Millán J (2011) Vision-based shared control for a BCI wheelchair. Int J Bioelectromagn 13(EPFL–ARTICLE–168977):20

    Google Scholar 

  18. Almeida L, Patrao B,Menezes P, Dias J (2014) Be the robot: Human embodiment in tele-operation driving tasks. In: 2014 RO-MAN: the 23rd IEEE international symposium on robot and human interactive communication. IEEE, pp 477–482

  19. Pioneer P3-DX (2016). http://www.mobilerobots.com/ResearchRobots/PioneerP3DX.aspx

  20. Intel NUC (2015). http://www.intel.com/content/www/us/en/nuc/overview.html

  21. OpenGL (2015). https://www.opengl.org

  22. G.R.A.S Microphones (2015). http://www.gras.dk

  23. NI USB-9234 (2015). http://sine.ni.com/nips/cds/view/p/lang/en/nid/204481

  24. Nakadai K, Takahashi T, Okuno HG, Nakajima H, Hasegawa Y, Tsujino H (2010) Design and implementation of robot audition system ‘HARK’-open source software for listening to three simultaneous speakers. Adv Robot 24(5–6):739

    Article  Google Scholar 

  25. Nakamura K, Nakadai K, Asano F, Ince G (2011) Intelligent sound source localization and its application to multimodal human tracking. In: 2011 IEEE/RSJ international conference on Intelligent Robots and Systems. IEEE, pp 143–148

  26. Okuno HG, Nakadai K, Takahashi T, Takeda R, Nakamura K, Mizumoto T, Yoshida T, Lim A, Otsuka T, Nagira K, Itohara T, Bando Y (2014) Hark document version 2.1.0. Tech. rep., Kyoto University

  27. Nakadai K, Ince G, Nakamura K, Nakajima H (2012) Robot audition for dynamic environments. In: 2012 IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC). IEEE, pp 125–130

  28. Nakamura K, Nakadai K, Asano F, Hasegawa Y, Tsujino H (2009) Intelligent sound source localization for dynamic environments. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009. IROS 2009. IEEE, pp 664–669

  29. Elmenreich W (2002) An introduction to sensor fusion. Tech. report, Vienna University of Technology, Austria 502:1–28

  30. Eickeler S, Kosmala A, Rigoll G (1998) Hidden markov model based continuous online gesture recognition. In: Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No. 98EX170), 1998. Proceedings, vol 2. IEEE, pp 1206–1208

  31. Fine S, Singer Y, Tishby N (1998) The hierarchical hidden Markov model: analysis and applications. Machi Learn 32(1):41

    Article  Google Scholar 

  32. Stephens M (1969) Tests for the von Mises distribution. Biometrika 56(1):149

    Article  Google Scholar 

  33. OptiTrack (2016). http://optitrack.com

Download references

Author information

Authors and Affiliations

  1. Mathworks Inc., Natick, USA

    Jianhao Du

  2. Department of Electrical and Computer Engineering, University of Louisville, Louisville, USA

    Ha Manh Do

  3. School of Electrical and Computer Engineering, Oklahoma State University, Stillwater, USA

    Weihua Sheng

Authors
  1. Jianhao Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ha Manh Do
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weihua Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weihua Sheng.

Ethics declarations

Conflict of interest

This material is based upon work supported by the National Science Foundation under Grant Nos. CISE/IIS 1427345, CISE/IIS 1910993, EHR/DUE 1928711, and CISE/IIS/SCH 1838808. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.This project is also partially supported by the Natural Science Foundation of China-Shenzhen Basic Research Center Project No. U1713216 and the Open Research Project of the State Key Laboratory of Industrial Control Technology, Zhejiang University, China (No. ICT20026). The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Du, J., Do, H.M. & Sheng, W. Human–Robot Collaborative Control in a Virtual-Reality-Based Telepresence System. Int J of Soc Robotics 13, 1295–1306 (2021). https://doi.org/10.1007/s12369-020-00718-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-020-00718-w

Keywords

Search

Navigation