Skip to main content
SpringerLink
Log in

3DGAM: using 3D gesture and CAD models for training on mixed reality remote collaboration

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

As Virtual Reality(VR), Augmented Reality(AR), Mixed Reality(MR) technology becomes more accessible, it is important to explore VR/AR/MR technologies that can be used for remote collaboration on physical tasks. Previous research has shown that gesture-based interaction is intuitive and expressive for remote collaboration, and using 3D CAD models can provide clear instructions for assembly tasks. In this paper, therefore, we describe a new MR remote collaboration system which combines the use of gesture and CAD models in a complementary manner. The prototype system enables a remote expert in VR to provide instructions based on 3D gesture and CAD models (3DGAM) for a local worker who uses AR to see these instructions. Using this interface, we conducted a formal user study to explore the effect of sharing 3D gesture and CAD models in an assembly training task. We found that the combination of 3D gesture and CAD models can improve remote collaboration on an assembly task with respect to the performance time and user experience. Finally, we provide some conclusions and directions for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig.  11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

Notes

  1. https://www.vive.com

  2. https://www.microsoft.com/en-us/hololens

  3. http://www.wampserver.com.

  4. https://github.com/microsoft/MixedRealityToolkit-Unity.

  5. https://docs.unity3d.com/Manual/AssetBundles-Workflow.html.

  6. https://im.qq.com.

  7. https://www.wechat.com.

  8. https://webrtc.org.

References

  1. Al-Khafajiy M, Baker T, Chalmers C, Asim M, Kolivand H, Fahim M, Waraich A (2019) Remote health monitoring of elderly through wearable sensors. Multimed Tools Appl 78(17):1–26

    Article  Google Scholar 

  2. Bai H, Prasanth S, Jing Y, Mark B (2020). A User Study on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture Sharing. 1–13. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM CHI ‘20, April 25–30, 2020, Honolulu, HI, USA.

  3. Brooke J (1996). SUS -- a quick and dirty usability scale

  4. Burova A, Mäkelä J, Hakulinen J, Keskinen T, Heinonen H, Siltanen S, Turunen M (2020). “Utilizing VR and Gaze Tracking to Develop AR Solutions for Industrial Maintenance.” (278):1–13. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM CHI ‘20, April 25–30, 2020, Honolulu, HI, USA

  5. Chen X, Gao S, Guo S (2012) Bai J (2012) a flexible assembly retrieval approach for model reuse[J]. Comput Aided Des 44(6):554–574

    Article  Google Scholar 

  6. Chénéchal L M, Duval T, Gouranton V, Royan J, Royan B (2016). Vishnu: virtual immersive support for HelpiNg users an interaction paradigm for collaborative remote guiding in mixed reality. In 2016 IEEE 3rd VR international workshop on collaborative virtual environments, 3DCVE 2016.

  7. D’Angelo S, Gergle D (2018). An Eye For Design: Gaze Visualizations for Remote Collaborative Work. Proceedings of the 2018 CHI conference on human factors in computing systems. ACM, 2018: 349, 1–12

  8. De Pace F, Federico M, Sanna A, Zappia D (2019) A comparison between two different approaches for a collaborative mixed-virtual environment in industrial maintenance. Frontiers in Robotics and AI 6:18

    Article  Google Scholar 

  9. Elvezio C, Sukan M, Feiner S, Tversky B (2015) Interactive visualizations for Monoscopic eyewear to assist in manually orienting objects in 3D, 2015 IEEE international symposium on mixed and augmented reality. IEEE 2015:180–181

    Google Scholar 

  10. Elvezio C, Sukan M, Oda O, Feiner S, Tversky B (2017) Remote collaboration in AR and VR using virtual replicas. In Association for Computing Machinery (ACM) SIGGRAPH 13:1–2

    Google Scholar 

  11. Ens B, Lanir J, Tang A, Bateman S, Lee G, Piumsomhoon T, Bilinghusrt M (2019) Revisiting collaboration through mixed reality: the evolution of groupware. International Journal of Human-Computer Studies 131:81–98

    Article  Google Scholar 

  12. García-Pereira I, Portalés C, Gimeno J, Casas S (2019) A collaborative augmented reality annotation tool for the inspection of prefabricated buildings. Multimed Tools Appl 79(4):1–19

    Google Scholar 

  13. Gonzalez-Franco M, Pizarro R, Cermeron J, Li K, Thorn J, Hutabarat W, Tiwari A, Bermell-Garcia P (2017) Immersive mixed reality for manufacturing training. Frontiers in robotics and AI 2017(4: 3):1–8

    Google Scholar 

  14. Gupta K, Lee G, Billinghurst M (2016) Do You See What I See? The Effect of Gaze Tracking on Task Space Remote Collaboration. IEEE Trans Vis Comput Graph 22:2413

    Article  Google Scholar 

  15. Gurevich P, Lanir J, Cohen B (2015) Design and implementation of teleadvisor: a projection-based augmented reality system for remote collaboration. Computer Supported Cooperative Work (CSCW) 24(6):527–562

    Article  Google Scholar 

  16. Harms C, Biocca F (2004). Internal consistency and reliability of the networked minds measure of social presence[J]

  17. Hinderks A, Schrepp M, Mayo FJD, Escalona MJ, Thomaschewski J (2019) Developing a UX KPI based on the user experience questionnaire. Computer Standards & Interfaces 65:38–44

    Article  Google Scholar 

  18. Huang R, Zhang S, Bai X, Xu C, Huang B (2015) An effective subpart retrieval approach of 3D CAD models for manufacturing process reuse. Comput Ind 67:38–53

    Article  Google Scholar 

  19. Kim S, Lee G, Huang W, Kim H, Woo W, Billinghurst M (2019). Evaluating the Combination of Visual Communication Cues for HMD-based Mixed Reality Remote Collaboration, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems ACM, 173

  20. Kritzler M, Murr M, Michahelles F (2016) Remotebob: support of on-site workers via a telepresence remote expert system, proceedings of the 6th international conference on the internet of things. ACM 2016:7–14

    Google Scholar 

  21. Oda O , Elvezio C , Sukan M, Feiner S, Tversky B (2015). Virtual replicas for remote assistance in virtual and augmented reality, proceedings of the 28th annual ACM symposium on user Interface software and technology\

  22. Petrangeli S, Pauwels D, Hooft JVD, Ziak M, Slowack J, Wauters T, Turck FD (2019) A scalable webrtc-based framework for remote video collaboration applications. Multimed Tools Appl 78:7419–7452

    Article  Google Scholar 

  23. Piumsomboon T, Dey A, Ens B, Lee G, Billinghurst M (2017). CoVAR: Mixed-Platform Remote Collaborative Augmented and Virtual Realities System with Shared Collaboration Cues, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). IEEE, 218–219

  24. Piumsomboon T, Dey A, Ens B, Lee G, Billinghurst M (2019) The effects of sharing awareness cues in collaborative mixed reality[J]. Frontiers in Robotics and AI 2019:6–5

    Google Scholar 

  25. Princle A, Campbell A G, Hutka S, Torrasso A, Couper C, Strunden F, Jast K, Croly R, Quigley R, Mckiernan R, Sweeney P, Keane M T (2018). Using an industry-ready AR HMD on a real maintenance task: AR benefits performance on certain task steps more than others, IEEE international symposium on mixed and augmented reality adjunct (ISMAR-adjunct). IEEE, 236-241

  26. Schrepp M (2015). User experience questionnaire handbook. All you need to know to apply the UEQ successfully in your project

  27. Sukan M, Elvezio C, Feiner S, Tversky B (2016). Providing assistance for orienting 3D objects using monocular eyewear, proceedings of the 2016 symposium on spatial user interaction. ACM, 89-98

  28. Tang A, Boyle M, Greenberg S (2004). Display and presence disparity in mixed presence groupware[C]//proceedings of the fifth conference on Australasian user interface-volume 28. Australian computer society, Inc., 73-82

  29. Teo T, Lawrence L, Lee G A, Billinghurst M, Adcock M (2019). Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction[C]//Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems ACM, 201

  30. Villanueva A, Zhu Z, Liu Z, Peppler K, Redick T, Ramani K (2020). Meta-AR-App: An Authoring Platform for Collaborative Augmented Reality in STEM Classrooms. (2): 1–14. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM CHI ‘20, April 25–30, 2020, Honolulu, HI, USA

  31. Wang P, Bai X, Billinghusrt M, Zhang S, Han D, Sun M, Wang Z, Lv H, Han S (2020). Haptic feedback helps me ? A VR-SAR remote collaborative system with tangible interaction. Int J Hum Comput Interact : 1–16. https://doi.org/10.1080/10447318.2020.1732140.

  32. Wang S, Parsons M, Stone-McLean J, Rogers P, Boyd S, Hoover K, Meruvia-Pastor O, Gong M, Smith A (2017) Augmented reality as a telemedicine platform for remote procedural training[J]. Sensors 17(10):2294

    Article  Google Scholar 

  33. Wang P, Zhang S, Bai X, Billinghurst M, He W, Sun M, Chen Y, Lv H, Ji H (2019) 2.5 DHANDS: a gesture-based MR remote collaborative platform. Int J Adv Manuf Technol 102(5–8):1339–1353

    Article  Google Scholar 

  34. Wang P, Zhang S, Bai X, Billinghurst M, He W, Wang S. Zhang X, Du J, Chen Y (2019). Head pointer or eye gaze: which helps more in MR remote collaboration. the 26th IEEE Conference on Virtual Reality and 3D User Interfaces: 1–2

  35. Wang P, Zhang S, Bai X, Billinghurst M, Zhang L, Wang S, Han D, Lv H, Yan Y (2019) A gesture- and head-based multi-modal interaction platform for MR remote collaboration. Int J Adv Manuf Technol 105:3031–3043

    Article  Google Scholar 

  36. Wang P, Zhang S, Billinghurst M, Bai X, He W, Wang S, Sun M, Zhang S (2019). A comprehensive survey of AR/MR-based co-Design in Manufacturing. Engineering with Computers: 1–24

  37. Wu T Y, Gong J, Seyed T, Yang X D (2019). Proxino: enabling prototyping of virtual circuits with physical proxies. UIST 2019 - proceedings of the 32nd annual ACM symposium on user Interface software and technology(UIST): 121–32

  38. Young J, Langlotz T, Cook M, Regenbrecht H (2019) Immersive Telepresence and remote collaboration using Mobile and wearable devices. IEEE Trans Vis Comput Graph 25(5):1908–1918

    Article  Google Scholar 

Download references

Acknowledgments

This research was financially sponsored by the civil aircraft special project (MJZ-2017-G73), Dongguan Science and Technology Equipment Project (KZ2018-05), National Key R&D Program of China (2019YFB1703800),Natural Science Basic Research Plan in Shaanxi Province of China (2016JM6054), and 111 project (B13044). Besides, we would like to appreciate Professor Mark Billinghurst for carefully checking the English of an early version of the paper. Additionally, we would like to thank Yuming Zheng for donating the water pump used in our research, Dechuan Han and Sili Wei for developing and improving the algorithm of sharing gestures, respectively, Jie Zhang, Guangyao Xu and Xu Zhang for preparing some supplementary materials.

Author information

Author notes

  1. Peng Wang, Xiaoliang Bai and Mark Billinghurst contributed equally to this work.

Authors and Affiliations

  1. Cyber-Physical Interaction Lab, Northwestern Polytechnical University, Xi’an, China

    Peng Wang, Xiaoliang Bai, Mark Billinghurst, Shusheng Zhang, Sili Wei, Guangyao Xu, Weiping He & Jie Zhang

  2. Empathic Computing Lab, University of South Australia, Mawson Lakes, Australia

    Mark Billinghurst

Authors
  1. Peng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoliang Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mark Billinghurst
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shusheng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sili Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Guangyao Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Weiping He
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xiangyu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jie Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Peng Wang, Xiaoliang Bai or Mark Billinghurst.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

(MP4 10,427 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, P., Bai, X., Billinghurst, M. et al. 3DGAM: using 3D gesture and CAD models for training on mixed reality remote collaboration. Multimed Tools Appl 80, 31059–31084 (2021). https://doi.org/10.1007/s11042-020-09731-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09731-7

Keywords

Search

Navigation