Skip to main content
SpringerLink
Log in

Design and Implementation of TeleAdvisor: a Projection-Based Augmented Reality System for Remote Collaboration

  • Published:
Computer Supported Cooperative Work (CSCW) Aims and scope Submit manuscript

Abstract

TeleAdvisor is a versatile projection-based augmented reality system designed for remote collaboration. It allows a remote expert to naturally guide a local user in need of assistance in carrying out physical tasks around real-world objects. The system consists of a small projector and two cameras mounted on top of a tele-operated robotic arm at the worker’s side, and an interface to view the camera stream, control the point-of-view and gesture using projected annotations at the remote expert’s side. TeleAdvisor provides a hands-free, mobile, low-cost solution that supports gesturing by the remote expert while minimizing the cognitive overhead of the local worker. We describe the challenges, design considerations and implementation details of the two phases of the TeleAdvisor prototype, as well as its evaluation and deployment at an industrial manufacturing center. We summarize our understandings from our experiences during the project and discuss the general implications for design of augmented reality remote collaboration systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10

Similar content being viewed by others

Notes

  1. For MicroVision ShowWX+HDMI these are especially noticeable in top left and right corners.

  2. The second video stream is transmitted for the sake of depth map calculation at the Helper’s workstation.

  3. This caused several servos to burn out.

References

  • Alem, Leila, Franco Tecchia, and Weidong Huang, (2011). HandsOnVideo: Towards a Gesture based Mobile AR System for Remote Collaboration. In Recent Trends of Mobile Collaborative Augmented Reality Systems. New York: Springer, pp. 135–148. doi:10.1007/978-1-4419-9845-3

  • Anderson, Richard J., Crystal Hoyer, Steven A. Wolfman, and Ruth Anderson. (2004). A study of digital ink in lecture presentation. In Proceedings of the SIGCHI conference on Human factors in computing systems. New York: ACM Press, pp. 567–574. doi: 10.1145/985692.985764

  • Azuma, Ronald. (1997). A survey of augmented reality. Presence: Teleoperators and Virtual Environments, vol. 6, no. 4, pp. 355–385. doi:10.1.1.30.4999

    Article  Google Scholar 

  • Azuma, Ronald, Yohan Baillot, Reinhold Behringer, Steven Feiner, Simon Julier, and Blair MacIntyre (2001). Recent advances in augmented reality. IEEE Computer Graphics and Applications, vol. 21, no. 6, pp. 34–47. doi:10.1109/38.963459

    Article  Google Scholar 

  • Baird, Kevin M., and Woodrow Barfield (1999). Evaluating the effectiveness of augmented reality displays for a manual assembly task. Virtual Reality. vol 4, no. 4, pp. 250–259. doi:10.1007/BF01421808

    Article  Google Scholar 

  • Bauer, Martin, Gerd Kortuem, and Zary Segall. (1999). Where are you pointing at? A study of remote collaboration in a wearable videoconference system. In Third International Symposium on Wearable Computers, San-Fransisco, CA, 18–19 October, 1999. Washington, DC: IEEE computer society, pp. 151–158. doi:10.1109/ISWC.1999.806696

  • Billinghurst, Mark, and Hirokazu Kato (2002). Collaborative augmented reality. Communications of the ACM. vol. 45, no. 7, pp. 64–70. doi:10.1145/514236.514265

    Article  Google Scholar 

  • Birchfield, Stan, and Carlo Tomasi (1998). A pixel dissimilarity measure that is insensitive to image sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 20, no. 4, pp. 401–406. doi:10.1109/34.677269

    Article  Google Scholar 

  • Birnholtz, Jeremy, Abhishek Ranjan, and Ravin Balakrishnan (2010). Providing Dynamic Visual Information for Collaborative Tasks: Experiments With Automatic Camera Control. Human-Computer Interaction. vol. 25, no. 4, pp. 261–287. doi:10.1080/07370024.2010.500146

    Article  Google Scholar 

  • Bradski, Gary, and Adrian Kaehler (2008). Learning OpenCV: Computer Vision with the OpenCV Library. Sebastopol, CA: OReilly Media Inc. doi:10.1109/MRA.2009.933612

    Google Scholar 

  • Brown, Michael, Aditi Majumder, and Ruigang Yang (2005). Camera-based calibration techniques for seamless multiprojector displays. IEEE Transactions on Visualization and Computer Graphics, vol. 11, no. 2, pp. 193–206

    Article  Google Scholar 

  • Chen, Jessie YC, Ellen C. Haas, and Michael J. Barnes (2007). Human performance issues and user interface design for teleoperated robots. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, vol. 37, no. 6, pp. 1231–1245. doi:10.1109/TSMCC.2007.905819

  • Darken, Rudolph P., and Barry Peterson (2001). Spatial Orientation, wayfinding and representation. In K. M. Stanney (Ed.): Handbook of Virtual Environment Technology. Mahwah, New Jersey: Lawrence Erlbaum Associates, pp. 493–518

    Google Scholar 

  • Flagg, Matthew, and James M. Rehg (2006). Projector-Guided Painting. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (UIST 2006). New York: ACM Press. pp. 235–244. doi:10.1145/1166253.1166290

  • Fong, Terrence, and Charles Thorpe. (2001). Vehicle teleoperation interfaces. Autonomous Robots, vol. 11, no. 1., pp 9–18. doi:10.1023/A:1011295826834

    Article  MATH  Google Scholar 

  • Fussell, Susan R., Leslie D. Setlock, and Robert E. Kraut. (2003). Effects of head-mounted and scene-oriented video systems on remote collaboration on physical tasks. In Proceedings of the Conference on Human Factors in Computing Systems - CHI′03, pp. 513–520. New-York: ACM Press. doi:10.1145/642611.642701

  • Fussell, Susan R., Leslie D. Setlock, Jie Yang, Jiazhi Ou, Elizabeth Mauer, and Adam DI Kramer. (2004). Gestures Over Video Streams to Support Remote Collaboration on Physical Tasks. Human-Computer Interaction. vol. 19, no. 3, pp. 273–309. doi:10.1207/s15327051hci1903_3

  • Gauglitz, Steffen, Cha Lee, Matthew Turk, and Tobias Höllerer. (2012). Integrating the physical environment into mobile remote collaboration. Proceedings of the 14th International Conference on Human-Computer Interaction with Mobile Devices and Services - MobileHCI′12. New-York: ACM Press. pp. 241–250. doi:10.1145/2371574.2371610

  • Gauglitz, Steffen, Benjamin Nuernberger, Matthew Turk, and Tobias Höllerer (2014). World-stabilized annotations and virtual scene navigation for remote collaboration. Proceedings of the 27th annual ACM symposium on User interface software and technology. New-York: ACM Press. pp. 449–459.

  • Gergle, Darren, Robert E. Kraut, and Susan R. Fussell. (2013). Using Visual Information for Grounding and Awareness in Collaborative Tasks. Human-Computer Interaction, vol. 28, no. 1, pp. 1–39. doi:10.1080/07370024.2012.678246

    Google Scholar 

  • Gurevich, Pavel, Joel Lanir, Benjamin Cohen, and Ran Stone. (2012). TeleAdvisor: a versatile augmented reality tool for remote assistance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New-York: ACM Press. pp. 619–622. doi:10.1145/2207676.2207763

  • Hartley, Richard, and Andrew Zisserman (2003). Multiple view geometry in computer vision. New York: Cambridge university press.

    Google Scholar 

  • Henderson, Steven J., and Steven Feiner. (2009). Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret. In Science and Technology Proceedings - IEEE 2009 International Symposium on Mixed and Augmented Reality, ISMAR 2009. Washington DC: IEEE Computing Society, pp. 135–144. doi:10.1109/ISMAR.2009.5336486

  • Henderson, Steven J., and Steven Feiner (2011). Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Transactions on Visualization and Computer Graphics, vol 17, no. 10, pp. 1355–1368. doi:10.1109/TVCG.2010.245

    Article  Google Scholar 

  • Hirschmüller, Heiko (2008). Stereo processing by semiglobal matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 2., pp. 328–341. doi:10.1109/TPAMI.2007.1166

    Article  Google Scholar 

  • Huang, Weidong, and Leila Alem (2011). Supporting hand gestures in mobile remote collaboration: a usability evaluation. In Proceedings of the 25th BCS Conference on Human-Computer Interaction. Swinton, UK: British Computer Society. pp. 211–216.

  • Huang, Weidong, Leila Alem, and Franco Tecchia. (2013). HandsIn3D: supporting remote guidance with immersive virtual environments. In Human-Computer Interaction--INTERACT 2013 lecture notes in computer science. vol. 8117, pp. 70–77. Berlin: Springer

  • Ishii, Hiroshi, Eran Ben-Joseph, John Underkoffler, Luke Yeung, Dan Chak, Zahra Kanji, and Ben Piper. (2002). Augmented urban planning workbench: overlaying drawings, physical models and digital simulation. In Proceedings of Ismar′02.The first International Symposium on Mixed and Augmented Reality. Washington DC: IEEE Computer Society, pp. 203–211 doi:10.1109/ISMAR.2002.1115090

  • Junuzovic, Sasa, Kori Inkpen, Tom Blank, and Anoop Gupta. (2012). IllumiShare: sharing any surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: ACM Press, pp. 1919–1928. doi:10.1145/2207676.2208333

  • Kirk, David, and Danae Stanton Fraser (2006). Comparing remote gesture technologies for supporting collaborative physical tasks. In Proceedings of the SIGCHI conference on Human Factors in computing systems. New York: ACM Press, pp. 1191–1200.

  • Kirk, David, Andy Crabtree, and Tom Rodden (2005). Ways of the Hands. In Proceedings of the ninth european conference on Computer-Supported Cooperative Work 18–22 September, Paris, France. pp. 1–22. Springer Netherlands.

  • Kirk, David, Tom Rodden, and Danaë Stanton Fraser (2007). Turn it THIS way: grounding collaborative action with remote gestures. In Proceedings of ACM CHI 2007 Conference on Human Factors in Computing Systems. New York: ACM Press, pp. 1039–1048. doi:10.1145/1240624.1240782

  • Kraut, Robert E., Susan R. Fussell, and Jane Siegel (2003). Visual Information as a Conversational Resource in Collaborative Physical Tasks. Human-Computer Interaction. vol. 18, no. 1, pp. 13–59. doi:10.1207/S15327051HCI1812_2

    Article  Google Scholar 

  • Kurata, Takeshi, Nobuchika Sakata, Masakatsu Kourogi, Hideaki Kuzuoka, and Mark Billinghurst. (2004). Remote collaboration using a shoulder-worn active camera/laser. Eighth International Symposium on Wearable Computers, vol 1. Washington DC: IEEE Computer Society, pp. 62–69. doi:10.1109/ISWC.2004.37

  • Kuzuoka, Hideaki, Shinya Oyama, Keiichi Yamazaki, Kenji Suzuki, and Mamoru Mitsuishi. (2000). GestureMan: a mobile robot that embodies a remote instructor’s actions. Proceedings of the ACM conference on Computer Supprted Cooperative Work. New York: ACM Press, pp. 155–162. doi:10.1145/358916.358986

  • Lanir, Joel, Ran Stone, Benjamin Cohen, and Pavel Gurevich (2013). Ownership and Control of Point of View in Remote Assistance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI′13. New York: ACM Press, pp. 2243–2252. doi:10.1145/2470654.2481309

  • Linder, Natan, and Pattie Maes. (2010). LuminAR: Portable Robotic Augmented Reality Interface Design and Prototype. In Adjunct proceedings of the 23nd annual ACM symposium on User interface software and technology. New York: ACM Press, pp. 395–396. doi:10.1145/1866218.1866237

  • Machino, Tamotsu, Satoshi Iwaki, Hiroaki Kawata, Yoshimasa Yanagihara, Yoshito Nanjo, and Ken-ichiro Shimokura (2006). Remote-collaboration system using mobile robot with camera and projector. In Proceedings - IEEE International Conference on Robotics and Automation 2006. Washington DC: IEEE Computer Society, pp. 4063–4068. doi:10.1109/ROBOT.2006.1642326

  • Majumder, Aditi, Zhu He, Herman Towles, and Greg Welch. (2000). Achieving color uniformity across multi-projector displays. Proceedings Visualization 2000. VIS 2000. Washington DC: IEEE Computer Society, pp 117–124. doi:10.1109/VISUAL.2000.885684

  • Medioni, Gerard, and Ramakant Nevatia (1985). Segment-based stereo matching. Computer Vision, Graphics, and Image Processing. vol 31, no 1., pp. 2–18. doi:10.1016/S0734-189X(85)80073-6

    Article  Google Scholar 

  • Mistry, Pranav, Pattie Maes, and Liyan Chang (2009). WUW - Wear Ur World: A Wearable Gestural Interface. In CHI′09 Extended Abstracts on Human Factors in Computing Systems. New York: ACM Press, pp. 4111–4116. doi:10.1145/1520340.1520626

  • Motokawa, Yoichi, and Hideo Saito (2007). Support system for guitar playing using augmented reality display. In Proceedings - ISMAR 2006: Fifth IEEE and ACM International Symposium on Mixed and Augmented Reality. Washington DC: IEEE Computer Society, pp. 243–244. doi:10.1109/ISMAR.2006.297825

  • Murphy, Robin Roberson (2004). Human-robot interaction in rescue robotics. IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews. vol 34. no. 2, pp 138–153. doi:10.1109/TSMCC.2004.826267

    Article  Google Scholar 

  • Neumann, Ulrich, and Anthony Majoros (1998). Cognitive, performance, and systems issues for augmented reality applications in manufacturing and maintenance. Proceedings. IEEE 1998 Virtual Reality Annual International Symposium. Washington DC: IEEE Computer Society, pp. 4–11. doi:10.1109/VRAIS.1998.658416

  • O’Neill, Jacki, Stefania Castellani, Frederic Roulland, Nicolas Hairon, Cornell Juliano, and Liwei Dai (2011). From ethnographic study to mixed reality: a remote collaborative troubleshooting system. In Proceedings of the ACM 2011 conference on Computer supported cooperative work. New York: ACM Press, pp. 225–234. doi:10.1145/1958824.1958859

  • Ou, Jiazhi, Susan R. Fussell, Xilin Chen, Leslie D. Setlock, and Jie Yang (2003). Gestural communication over video stream: supporting multimodal interaction for remote collaborative physical tasks. In Proceedings of the 5th international conference on Multimodal interfaces. New York: ACM Press, pp. 242–249.

  • Ranjan, Abhishek, Jeremy P. Birnholtz, and Ravin Balakrishnan (2007). Dynamic shared visual spaces: experimenting with automatic camera control in a remote repair task. In Proceedings of ACM CHI 2007 Conference on Human Factors in Computing Systems. New York: ACM Press, pp. 1177–1186. doi:10.1145/1240624.1240802

  • Raskar, Ramesh, Michael S. Brown, Ruigang Yang, Wei-Chao Chen, Greg Welch, Herman Towles, Brent Scales, and Henry Fuchs (1999). Multi-projector displays using camera-based registration. In Visualization′99. Proceedings. Washington DC: IEEE Computer Society, pp. 161–522.

  • Sheridan, Thomas B. (2002). Humans and Automation: System Design and Research Issues. New York: John Wiley & Sons.

    Google Scholar 

  • Sodhi, Rajinder, Hrvoje Benko, and Andrew Wilson (2012). Lightguide: projected visualizations for hand movement guidance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: ACM Press, pp. 179–188. doi:10.1145/2207676.2207702

  • Tang, Anthony, Carman Neustaedter, and Saul Greenberg (2006). VideoArms: Embodiments for Mixed Presence Groupware. In People and Computers XX - Engage. London: Springer, pp. 85–102. doi:10.1007/978-1-84628-664-3_8

  • Tecchia, Franco, Leila Alem, and Weidong Huang (2012). 3D helping hands: a gesture based MR system for remote collaboration. Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry - VRCAI′12. New York: ACM Press, pp. 323–328. doi:10.1145/2407516.2407590

  • Tsai, Roger Y., and Reimar K. Lenz. (1988). Real time versatile robotics hand/eye calibration using 3D machine vision. In Robotics and Automation, 1988 Proceedings., IEEE International Conference on. Washington DC: IEEE Computer Society, pp. 554–561.

  • Wellner, Pierre (1993). Interacting with paper on the DigitalDesk. Communications of the ACM. vol. 36, no. 7, pp. 87–96. doi:10.1145/159544.159630

    Article  Google Scholar 

  • Woods, David D., James Tittle, Magnus Feil, and Axel Roesler (2004). Envisioning human-robot coordination in future operations. IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews, vol. 34, no. 2, pp. 210–218. doi:10.1109/TSMCC.2004.826272

    Article  Google Scholar 

  • Zhang, Zhengyou, Rachid Deriche, Olivier Faugeras, and Quang-Tuan Luong (1995). A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry. Artificial intelligence, vol. 78, no 1, pp. 87–119.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IBM Haifa Research Laboratory, Mt. Carmel, Haifa, 31905, Israel

    Pavel Gurevich & Benjamin Cohen

  2. University of Haifa, Mt. Carmel, Haifa, 31905, Israel

    Joel Lanir

Authors
  1. Pavel Gurevich
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joel Lanir
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Benjamin Cohen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joel Lanir.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gurevich, P., Lanir, J. & Cohen, B. Design and Implementation of TeleAdvisor: a Projection-Based Augmented Reality System for Remote Collaboration. Comput Supported Coop Work 24, 527–562 (2015). https://doi.org/10.1007/s10606-015-9232-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10606-015-9232-7

Keywords

Search

Navigation