Skip to main content
SpringerLink
Log in

Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Distributed collaborations between two or more participants on a task involving tangible artifacts (e.g., a machine, a patient, a tool) have become increasingly common in recent years due to rapid development in information and communication technologies. In this paper we focus on a specific type of remote-collaboration system where a remote helper guides a local worker using audio communication and hand gestures to perform a repair or a maintenance task. An established ICT approach to supporting this type of collaboration is to provide a shared visual space and some forms of remote gesture. The shared space typically consists of a video capture of the remote workspace which is then displayed on a 2D screen. However, this type of approach has its limitations. Firstly, it does not provide the helper with sufficient understanding of the spatial relationships between objects in the remote workspace. Secondly, it does not allow the helper to gesture in 3D. In an attempt to address these issues, we propose a Mixed Reality multimodal system that improves on previous 2D systems by introducing 3D real-time capturing and rendering of both the remote workspace and the helping hands and by creating a 3D shared visual space as a result of co-locating the remote workspace with the helping hands. In this system, we explore the possibility of increasing the feeling of immersion and co-presence by using head tracking, stereoscopic rendering, inter-occlusion handling and virtual shadowing. In this paper, we introduce HandsIn3D, a system that has been developed for the purpose of the proof of concept. We also present the results of experiments to verify the feasibility of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Clark HH, Brennan SE (1991) Grounding in communication. In perspectives on socially shared cognition. American Psychological Association, Washington, DC

    Google Scholar 

  2. Liu J, Kavakli M (2010) A survey of speech-hand gesture recognition for the development of multimodal interfaces in computer games. In: IEEE international conference on multimedia and expo (ICME), pp 1564–1569

  3. Fussell SR, Setlock LD, Yang J, Ou J, Mauer E, Kramer ADI (2004) Gestures over video streams to support remote collaboration on physical tasks. Hum Comput Interact 19:273–309

    Article  Google Scholar 

  4. Kirk DS, Stanton Fraser D (2006) Comparing remote gesture technologies for supporting collaborative physical tasks. In: ACM human factors in computing systems, pp 1191–1200

  5. Ou J, Fussell SR, Chen X, Setlock LD, Yang J (2003) Gestural communication over video stream: supporting multimodal interaction for remote collaborative physical tasks. In: Proceedings of the 5th conference on multimodal interfaces, pp 242–249

  6. Kuzuoka H, Kosaka J, Yamazaki K, Suga Y, Yamazaki A, Luff P, Heath C (2004) Mediating dual ecologies. In: ACM conference on computer supported cooperative work, pp 477–486

  7. Kuzuoka H (1992) Spatial workspace collaboration: a SharedView video support system for remote collaboration capability. In: ACM human factors in computing systems, pp 533–540

  8. Yamashita N, Kaji K, Kuzuoka H, Hirata K (2011) Improving visibility of remote gestures in distributed tabletop collaboration. In: ACM conference on computer supported cooperative work, pp 95–104

  9. Alem L, Tecchia F, Huang W (2011) HandsOnVideo: towards a gesture based mobile AR system for remote collaboration. In: Alem L, Huang W (eds) Recent trends of mobile collaborative augmented reality. Springer, New York, pp 127–138

  10. Huang W, Alem L (2011) Supporting hand gestures in mobile remote collaboration: a usability evaluation. In: Proceedings of the 25th BCS conference on human computer interaction

  11. Kraut RE, Gergle D, Fussell SR (2002) The Use of visual information in shared visual spaces: informing the development of virtual co-presence. In: ACM conference on computer supported cooperative work, pp 31–40

  12. Mortensen J, Vinayagamoorthy V, Slater M, Steed A, Lok B, Whitton MC (2002) Collaboration in tele-immersive environments. In: EGVE’02, pp 93–101

  13. Schuchardt P, Bowman DA (2007) The benefits of immersion for spatial understanding of complex underground cave systems. In: VRST’07, pp 121–124

  14. Mania K, Badariah S, Coxon M, Watten P (2010) Cognitive transfer of spatial awareness states from immersive virtual environments to reality. ACM Trans Appl Percept 7(2):9–14

    Article  Google Scholar 

  15. Huang W, Alem L, Tecchia F (2013) HandsIn3D: supporting remote guidance with immersive virtual environments. In: Human–computer interaction-INTERACT 2013. Lecture notes in computer science, vol 8117, pp. 70–77

  16. Gurevich P, Lanir J, Cohen B, Stone R (2011) TeleAdvisor: a versatile augmented reality tool for remote assistance. In: CHI’11, pp 619–622

  17. Alem L, Li J (2011) A study of gestures in a video-mediated collaborative assembly task. Adv Hum Comput Interact 2011. Article ID 987830. https://doi.org/10.1155/2011/987830

  18. Huang W, Alem L (2013) HandsinAir: a wearable system for remote collaboration on physical tasks. In: Proceedings of the 2013 conference on computer supported cooperative work companion (CSCW ’13), pp 153–156

  19. Huang W, Alem L (2013) Gesturing in the air: supporting full mobility in remote collaboration on physical tasks. J Univ Comput Sci 19(8):1158–1174

    Google Scholar 

  20. Sakata N, Kurata T, Kato T, Kourogi M, Kuzuoka H (2003) WACL: supporting telecommunications using-wearable active camera with laser pointer. In: Proceedings of the seventh international symposium on wearable computers, pp 53–56

  21. Kopácsi S, Kovács G, Nacsa J (2013) Some aspects of dynamic 3D representation and control of industrial processes via the Internet, Comput Ind. https://doi.org/10.1016/j.compind.2013.06.007. ISSN: 0166–3615

  22. Brown RA, Recker JC, West S (2011) Using virtual worlds for collaborative business process modeling. Bus Process Manag J 17(3):546–564

    Article  Google Scholar 

  23. Monahan T, McArdle G, Bertolotto M (2008) Virtual reality for collaborative e-learning. Comput Educ 50:1339–1353

    Article  Google Scholar 

  24. Higuch K, Yonetani R, Sato Y (2016) Can eye help you? Effects of visualizing eye fixations on remote collaboration scenarios for physical tasks. In: Proceedings of the 2016 CHI conference on human factors in computing systems (CHI’16), pp 5180–5190

  25. Gupta K, Lee G, Billinghurst M (2016) Do you see what i see? The effect of gaze tracking on task space remote collaboration. IEEE Trans Vis Comput Graph 22:2413–2422

    Article  Google Scholar 

  26. Otsuki M, Kawano T, Maruyama K, Kuzuoka H, Suzuki Y (2017) ThirdEye: simple add-on display to represent remote participant’s gaze direction in video communication. In: Proceedings of the 2017 CHI conference on human factors in computing systems (CHI’17). ACM, New York, pp 5307–5312

  27. Huang W, Alem L, Livingston MA (2013) Human factors in augmented reality environments. Springer, Berlin

    Book  Google Scholar 

  28. Gao G, Yamashita N, Hautasaari AMJ, Fussell SR (2015) Improving multilingual collaboration by displaying how non-native speakers use automated transcripts and bilingual dictionaries. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems (CHI’15), pp 3463–3472

  29. Nguyen DT, Fussell SR (2014) Verbal cues of involvement in dyadic same-culture and cross-culture instant messaging conversations. In: Proceedings of the 5th ACM international conference on collaboration across boundaries: culture, distance and technology (CABS’14), pp 41–50

  30. Maimone A, Fuchs H (2011) Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras. In: The IEEE international symposium on mixed and augmented reality (ISMAR), 26–29 Oct 2011, Basel, Switzerland

  31. Tecchia F, Alem L, Huang W (2012) 3D helping hands: a gesture based MR system for remote collaboration. In: Proceedings of the 11th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry (VRCAI’12), pp 323–328

  32. Pece F, Kautz J, Weyrich T (2011) Adapting standard video codecs for depth streaming. In: Proceedings of the 17th eurographics conference on virtual environments and third joint virtual reality (EGVE-JVRC’11), pp 59–66

  33. Bannò F, Gasparello P, Tecchia F, Bergamasco M (2012) Real-time compression of depth streams through meshification and valence-based encoding. In: Proceedings of the 11th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry (VRCAI ’12), pp 263–270

  34. Marino G, Gasparello P, Vercelli D, Tecchia F, Bergamasco M (2010) Network streaming of dynamic 3D content with on-line compression of frame data. In: VR 2010: pp 285–286

Download references

Author information

Authors and Affiliations

  1. Swinburne University of Technology, Melbourne, Australia

    Weidong Huang

  2. ArcSense, Sydney, Australia

    Leila Alem

  3. Scuola Superiore Sant’Anna, Pisa, Italy

    Franco Tecchia

  4. La Trobe University, Melbourne, Australia

    Henry Been-Lirn Duh

Authors
  1. Weidong Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leila Alem
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Franco Tecchia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Henry Been-Lirn Duh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weidong Huang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, W., Alem, L., Tecchia, F. et al. Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration. J Multimodal User Interfaces 12, 77–89 (2018). https://doi.org/10.1007/s12193-017-0250-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-017-0250-2

Keywords

Search

Navigation