Skip to main content
SpringerLink
Log in

An Augmented Reality-Based Method for Remote Collaborative Real-Time Assistance: from a System Perspective

  • Published:
Mobile Networks and Applications Aims and scope Submit manuscript

Abstract

To provide remote assistance to people more efficiently, an augmented reality (AR)-based method for remote real-time assistance for collaboration is proposed. This paper aims to reduce communication barriers and enhance the three-dimensional (3D) feel of immersive interactions. First, a multiplayer real-time video communication framework with WebRTC is built, which enables remote experts to observe a first-hand view of an operator’s site. Second, a shared cross-platform virtual whiteboard based on Canvas, WebSocket and Node.js is developed that enables remote experts to provide visual assistance, such as drawings or text, and adjust the position of the whiteboard for seamless integration with video. Last, the virtual assistance information provided by the remote experts is displayed on the screen of AR holographic glasses to enhance the assistance capability of the platform and enable an expert to explain to an operator how to correctly perform tasks. A hybrid tracking and registration technique based on natural features and gyroscopes is adopted to estimate the operator’s posture in real time to enable the virtual assistance information to be perfectly integrated with the real world at all times. An experimental analysis shows that this system is both practicable and stable and has broad application prospects in many fields.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Bajura M, Neumann U (1995) Dynamic registration correction in augmented-reality systems[C]. In: Virtual reality international symposium, 1995. Proceedings. IEEE, pp 189–196

  2. Hanna MG, Ahmed I, Nine J et al (2018) Augmented reality technology using Microsoft HoloLens in anatomic pathology [J]. Arch Pathol Lab Med 142(5):638–644

    Article  Google Scholar 

  3. Shenai MB, Dillavou M, Shum C et al (2011) Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance.[J]. Neurosurgery 68(2):200–207

    Google Scholar 

  4. Choi J, Yoon B, Jung C et al (2017) ARClassNote: augmented reality based remote education solution with tag recognition and shared hand-written note[C]. In: IEEE international symposium on mixed and augmented reality. IEEE, pp 303–309

  5. WebRTC Official Website. http://www.webrtc.org/. Accessed 25 June 2018

  6. Fette I, Melnikov A (2011) The WebSocket protocol. IETF internet draft, work in progress

  7. Halo Mini Official Website. http://www.shadowcreator.com/. Accessed 10 June 2018

  8. W3C WebRTC 1.0: Real-time communication between browsers. https://www.w3.org/TR/2018/CR-webrtc-20180621/. Accessed 28 June 2018

  9. IETF real-time communication in web-browsers (RTCWEB). http://datatracker.ietf.org/wg/rtcweb/. Accessed 28 June 2018

  10. Jang-Jaccard J, Nepal S, Celler B et al (2016) WebRTC-based video conferencing service for telehealth [J]. Computing 98(1-2):169–193

    Article  MathSciNet  Google Scholar 

  11. Ma LV, Kim J, Park S et al (2016) An efficient Session_Weight load balancing and scheduling methodology for high-quality telehealth care service based on WebRTC [J]. J Supercomput 72(10):3909–3926

    Article  Google Scholar 

  12. Santosgonzález I, Riverogarcía A, Molinagil J et al (2017) Implementation and analysis of real-time streaming protocols [J]. Sensors 17(4):846

    Article  Google Scholar 

  13. Johnston AB, Burnett DC (2012) WebRTC: APIs and RTCWEB protocols of the HTML5 real-time web [M]. Digital codex LLC

  14. Shen-hui C (2013) The study and implementation of NAT traversal technology. Nanjing University of Posts and Telecommunications

  15. Ringe S, Kedia R, Poddar A et al (2015) HTML5 based virtual whiteboard for real time interaction ☆[J]. Procedia Comput Sci 49(1):170–177

    Article  Google Scholar 

  16. Antunes M, Silva C, Barranca JA (2016) A Telemedicine application using WebRTC ☆ [J]. Procedia Comput Sci 100:414–420

    Article  Google Scholar 

  17. Pimentel V, Nickerson BG (2012) Communicating and displaying real-time data with WebSocket [J]. IEEE Internet Comput 16(4):45–53

    Article  Google Scholar 

  18. Tilkov S, Vinoski S (2010) Node.js: using JavaScript to build high-performance network programs [J]. IEEE Internet Comput 14(6):80–83

    Article  Google Scholar 

  19. Gabor D (1948) A new microscopic principle. [J]. Nature 161(4098):777

    Article  Google Scholar 

  20. Zeng F, Zhang X, Zhang J-p et al (2013) Holographic waveguide head-mounted display system design based on prisms-grating structure [J]. Acta Opt Sin 33(9):114–119

    Google Scholar 

  21. Anousaki GC, Kyriakopoulos KJ (1999) Simultaneous localization and map building for mobile robot navigation [J]. IEEE Robot Autom Mag 6(3):42–53

    Article  Google Scholar 

  22. Davison AJ, Reid ID, Molton ND et al (2007) MonoSLAM: real-time single camera SLAM.[J]. IEEE Trans Pattern Anal Mach Intell 29(6):1052–1067

    Article  Google Scholar 

  23. Mur-Artal R, Montiel JMM, Tardós JD (2017) ORB-SLAM: a versatile and accurate monocular SLAM system [J]. IEEE Trans Robot 31(5):1147–1163

    Article  Google Scholar 

  24. Mirzaei FM, Roumeliotis SI (2007) A Kalman filter-based algorithm for IMU-camera calibration[C]. In: Ieee/rsj international conference on intelligent robots and systems. IEEE, pp 2427–2434

  25. Zhou S-l, Wu X-z, Liu G et al (2016) Integrated navigation method of monocular ORB-SLAM/INS [J]. J Chin Iner Techno 24(5):633–637

    Google Scholar 

  26. Sun L, Du J, Qin W (2015) Research on combination positioning based on natural features and gyroscopes for AR on Mobile phones[C]. In: International conference on virtual reality and visualization. IEEE, pp 301–307

  27. Bailey T, Nieto J, Guivant J et al (2006) Consistency of the EKF-SLAM algorithm[C]. In: Ieee/rsj international conference on intelligent robots and systems. IEEE, pp 3562–3568

  28. Klein G, Murray D (2009) Parallel tracking and mapping on a camera phone[C]. In: IEEE international symposium on mixed and augmented reality. IEEE Computer Society, pp 83–86

  29. Davis BS (1998) Using low-cost MEMS accelerometers and gyroscopes as strapdown IMUs on rolling projectiles[C]. In: Position location and navigation symposium, IEEE. IEEE, pp 594–601

  30. Anton D, Kurillo G, Yang AY et al (2017) Augmented telemedicine platform for real-time remote medical consultation[M]. In: MultiMedia modeling. Springer International Publishing, pp 77–89

  31. Hou L, Wang X, Truijens M (2012) Using augmented reality to facilitate piping assembly: an experiment-based evaluation[J]. J Comput Civ Eng 29(1):05014007

    Article  Google Scholar 

  32. Oyekan J, Prabhu V, Tiwari A et al (2017) Remote real-time collaboration through synchronous exchange of digitised human-workpiece interactions[J]. Futur Gener Comput Syst 67:83–93

    Article  Google Scholar 

  33. Mimaroğlu O (2014) Collaborative augmented reality[J]. Commun ACM 45(7):64–70

    Google Scholar 

  34. Kim Y, Hong S, Kim GJ (2017) Augmented reality-based remote coaching for fast-paced physical task[J]. Virtual Reality 6:1–12

    Google Scholar 

  35. Cidota M, Lukosch S, Datcu D et al (2016) Workspace awareness in collaborative AR using HMDS: a user study comparing audio and visual notifications. In: Proceedings of the 7th augmented human international conference. ACM Press, New York

  36. Gauglitz S, Nuernberger B, Turk M (2014) In touch with the remote world:remote collaboration with augmented reality drawings and virtual navigation[C]. In: Proceedings of the 20th ACM symposium on virtual reality software and technology. ACM Press, New York, pp 197–205

Download references

Funding

This work is supported by the National Natural Science Foundation of China Grant No. 61502294, the CERNET Innovation Project under Grant Nos. NGII20170513 and NGII20170206, and the IIOT Innovation and Development Special Foundation of Shanghai under Grant No. 2017-GYHLW- 01037.

Author information

Authors and Affiliations

  1. School of Computer Engineering and Science, Shanghai University, Shanghai, 200444, China

    Dikai Fang, Huahu Xu & Minjie Bian

  2. Shanghai Shang Da Hai Run Information System Co., Ltd, Shanghai, 200444, China

    Huahu Xu & Minjie Bian

  3. School of Computer and Information Engineering, Shanghai Polytechnic University, Shanghai, 201209, China

    Xiaoxian Yang

  4. Shanghai Key Laboratory of Intelligent Manufacturing and Robotics, Shanghai, 200444, China

    Xiaoxian Yang

Authors
  1. Dikai Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huahu Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoxian Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Minjie Bian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoxian Yang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fang, D., Xu, H., Yang, X. et al. An Augmented Reality-Based Method for Remote Collaborative Real-Time Assistance: from a System Perspective. Mobile Netw Appl 25, 412–425 (2020). https://doi.org/10.1007/s11036-019-01244-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11036-019-01244-4

Keywords

Search

Navigation