Émilie Fabre
Jun Rekimoto
ABSTRACT
This study explores the value of using virtual humans (VHs) to mask real entities within augmented reality (AR) interactive environments. We believe that one ultimate solution to bridge the gap between virtual and real worlds is to enable virtual entities to interact with the real world seamlessly and physically. However, unlike major approaches such as displaying haptics directly onto real users, letting virtual entities affect the real world physically is an underrepresented field in AR. To explore this approach, we employ reality actuators, here a robot arm, behind an AR VH as it may improve the sense of social presence and engagement in human-robot interaction systems. We developed an AR system where a VH is overlaid onto a robotic arm participating in a chess game. The preliminary results of our pilot study suggest the system’s significant potential to not only enhance the perceived social presence of VHs but also increase overall presence and user engagement, especially compared to situations where the mechanisms are visible without virtual concealment. We hope this work paves the way for more seamless reality experience between real and virtual worlds.
Supplemental Material
Available for Download
zip
AHs_KurokoParadigm_Suppmat.zip (59.1 MB)
.zip file containing a demo video (H.264, .mp4) as well as a .pdf file of the full statistical analysis.
- Samer Al Moubayed, Jonas Beskow, Gabriel Skantze, and Björn Granström. 2012. Furhat: A Back-Projected Human-Like Robot Head for Multiparty Human-Machine Interaction. In Cognitive Behavioural Systems, Anna Esposito, Antonietta M. Esposito, Alessandro Vinciarelli, Rüdiger Hoffmann, and Vincent C. Müller (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 114–130.Google Scholar
- Takafumi Aoki, Takashi Matsushita, Yuichiro Iio, Hironori Mitake, Takashi Toyama, Shoichi Hasegawa, Rikiya Ayukawa, Hiroshi Ichikawa, Makoto Sato, Takatsugu Kuriyama, Kazuyuki Asano, Toshihiro Kawase, and Itaru Matumura. 2005. Kobito: virtual brownies. In ACM SIGGRAPH 2005 Emerging Technologies (Los Angeles, California) (SIGGRAPH ’05). Association for Computing Machinery, New York, NY, USA, 11–es. https://doi.org/10.1145/1187297.1187309Google ScholarDigital Library
- Sahar Aseeri and Victoria Interrante. 2021. The Influence of Avatar Representation on Interpersonal Communication in Virtual Social Environments. IEEE Transactions on Visualization and Computer Graphics 2626 (2021), 1–1. Issue c. https://doi.org/10.1109/tvcg.2021.3067783Google ScholarCross Ref
- Ronald T Azuma. 1997. A survey of augmented reality. Presence: teleoperators & virtual environments 6, 4 (1997), 355–385.Google Scholar
- Jeremy N Bailenson and Jim Blascovich. 2004. Avatars. In Berkshire Encyclopedia of Human-Computer Interaction. Berkshire, Great Barrington, MA, USA, 64–68.Google Scholar
- Marios Bikos, Yuta Itoh, Gudrun Klinker, and Konstantinos Moustakas. 2015. An interactive augmented reality chess game using bare-hand pinch gestures. In 2015 International Conference on Cyberworlds (CW). IEEE, IEEE, Visby, Sweden, 355–358. https://doi.org/10.1109/CW.2015.15Google ScholarDigital Library
- Frank Biocca and Chad Harms. 2002. Defining and measuring social presence: Contribution to the networked minds theory and measure. Proceedings of PRESENCE 2002 (2002), 7–36.Google Scholar
- Pulkit Budhiraja, Rajinder Sodhi, Brett Jones, Kevin Karsch, Brian Bailey, and David Forsyth. 2015. Where’s My Drink? Enabling Peripheral Real World Interactions While Using HMDs. arxiv:1502.04744 [cs.HC]Google Scholar
- Yen-Fu Chen, Sylvia Janicki, 2020. A cognitive-based board game with augmented reality for older adults: Development and usability study. JMIR serious games 8, 4 (2020), e22007.Google Scholar
- David T. Coleman, Ioan A. Sucan, Sachin Chitta, and Nikolaus Correll. 2014. Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Study. JOSER - Journal of Software Engineering for Robotics 5 (2014), 3–16. https://doi.org/10.6092/JOSER_2014_05_01_P3Google ScholarCross Ref
- Bruce Damer, Richard Walker, Jeremy Judson, Jackie Dove, Steve DiPaola, Ali Ebtekar, Stasia McGehee, and Kate Reber. 1997. Avatars!; Exploring and Building Virtual Worlds on the Internet (1st ed.). Peachpit Press, USA.Google Scholar
- Chloe Eghtebas, Gudrun Klinker, Susanne Boll, and Marion Koelle. 2023. Co-Speculating on Dark Scenarios and Unintended Consequences of a Ubiquitous(Ly) Augmented Reality. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (Pittsburgh, PA, USA) (DIS ’23). Association for Computing Machinery, New York, NY, USA, 2392–2407. https://doi.org/10.1145/3563657.3596073Google ScholarDigital Library
- Steven Feiner. 1994. Redefining the user interface: Augmented reality. In Course Notes: Developing advanced virtual reality applications(Course Notes: Siggraph 1994, 21st International Conference on Computer Graphics and Interactive Techniques). Association for Computing Machinery, Orlando, FL, USA, 1–18.Google Scholar
- Jann Philipp Freiwald, Julius Schenke, Nale Lehmann-Willenbrock, and Frank Steinicke. 2021. Effects of Avatar Appearance and Locomotion on Co-Presence in Virtual Reality Collaborations. In Proceedings of Mensch Und Computer 2021 (Ingolstadt, Germany) (MuC ’21). Association for Computing Machinery, New York, NY, USA, 393–401. https://doi.org/10.1145/3473856.3473870Google ScholarDigital Library
- Makio Fukuda and Hisayoshi Horioka. 2013. Improvement in the fun of the board game by AR introduction (In the case of Japanese Board Game “Sugoroku”). In 2013 IEEE 2nd Global Conference on Consumer Electronics (GCCE). IEEE, IEEE, Tokyo, Japan, 334–338.Google ScholarCross Ref
- Tianyang Gao and Yuta Itoh. 2019. The Kuroko Paradigm: The Implications of Augmenting Physical Interaction with AR Avatars. In 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, Beijing, China, 26–27. https://doi.org/10.1109/ISMAR-Adjunct.2019.00022Google ScholarCross Ref
- Scott A Green, Mark Billinghurst, XiaoQi Chen, and J Geoffrey Chase. 2008. Human-robot collaboration: A literature review and augmented reality approach in design. International journal of advanced robotic systems 5, 1 (2008), 1.Google ScholarCross Ref
- Jens Grubert, Tobias Langlotz, Stefanie Zollmann, and Holger Regenbrecht. 2016. Towards pervasive augmented reality: Context-awareness in augmented reality. IEEE transactions on visualization and computer graphics 23, 6 (2016), 1706–1724.Google Scholar
- Sandra G. Hart. 2006. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 9 (Oct. 2006), 904–908. https://doi.org/10.1177/154193120605000909Google ScholarCross Ref
- Carrie Heater. 1992. Being there: The subjective experience of presence.Presence Teleoperators Virtual Environ. 1, 2 (1992), 262–271.Google Scholar
- Duy-Nguyen Ta Huynh, Karthik Raveendran, Yan Xu, Kimberly Spreen, and Blair MacIntyre. 2009. Art of defense: a collaborative handheld augmented reality board game. In Proceedings of the 2009 ACM SIGGRAPH Symposium on Video Games (, New Orleans, Louisiana,) (Sandbox ’09). Association for Computing Machinery, New York, NY, USA, 135–142. https://doi.org/10.1145/1581073.1581095Google ScholarDigital Library
- Masahiko Inami, Naoki Kawakami, and Susumu Tachi. 2003. Optical camouflage using retro-reflective projection technology. In The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings.IEEE, Tokyo, Japan, 348–349. https://doi.org/10.1109/ISMAR.2003.1240754Google ScholarCross Ref
- Brennan Jones, Yaying Zhang, Priscilla N. Y. Wong, and Sean Rintel. 2021. Belonging There: VROOM-ing into the Uncanny Valley of XR Telepresence. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, Article 59 (apr 2021), 31 pages. https://doi.org/10.1145/3449133Google ScholarDigital Library
- Shunichi Kasahara, Ryuma Niiyama, Valentin Heun, and Hiroshi Ishii. 2013. exTouch: spatially-aware embodied manipulation of actuated objects mediated by augmented reality. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction (Barcelona, Spain) (TEI ’13). Association for Computing Machinery, New York, NY, USA, 223–228. https://doi.org/10.1145/2460625.2460661Google ScholarDigital Library
- Keishirou Kataoka, Takuya Yamamoto, Mai Otsuki, Fumihisa Shibata, and Asako Kimura. 2019. A new interactive haptic device for getting physical contact feeling of virtual objects. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, IEEE, Osaka, Japan, 1323–1324. https://doi.org/10.1109/VR.2019.8797762.Google ScholarCross Ref
- Kangsoo Kim, Luke Boelling, Steffen Haesler, Jeremy Bailenson, Gerd Bruder, and Greg F. Welch. 2018. Does a Digital Assistant Need a Body? The Influence of Visual Embodiment and Social Behavior on the Perception of Intelligent Virtual Agents in AR. In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, Munich, Germany, 105–114. https://doi.org/10.1109/ISMAR.2018.00039Google ScholarCross Ref
- Sven Kratz and Fred Rabelo Ferriera. 2016. Immersed remotely: Evaluating the use of head mounted devices for remote collaboration in robotic telepresence. In 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, IEEE Press, New York, NY, USA, 638–645.Google ScholarDigital Library
- Isabel Leal, Krzysztof Choromanski, Deepali Jain, Avinava Dubey, Jake Varley, Michael Ryoo, Yao Lu, Frederick Liu, Vikas Sindhwani, Quan Vuong, Tamas Sarlos, Ken Oslund, Karol Hausman, and Kanishka Rao. 2023. SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention. arxiv:2312.01990 [cs.RO]Google Scholar
- Kwan Min Lee. 2004. Presence, explicated. Communication theory 14, 1 (2004), 27–50.Google Scholar
- Myungho Lee, Kangsoo Kim, Salam Daher, Andrew Raij, Ryan Schubert, Jeremy Bailenson, and Greg Welch. 2016. The wobbly table: Increased social presence via subtle incidental movement of a real-virtual table. In 2016 IEEE virtual reality (VR). IEEE, IEEE, Greenville, SC, USA, 11–17.Google Scholar
- Myungho Lee, Nahal Norouzi, Gerd Bruder, Pamela J. Wisniewski, and Gregory F. Welch. 2018. The physical-virtual table: exploring the effects of a virtual human’s physical influence on social interaction. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (Tokyo, Japan) (VRST ’18). Association for Computing Machinery, New York, NY, USA, Article 25, 11 pages. https://doi.org/10.1145/3281505.3281533Google ScholarDigital Library
- Yue Li, Eugene Ch’ng, and Sue Cobb. 2023. Factors Influencing Engagement in Hybrid Virtual and Augmented Reality. ACM Trans. Comput.-Hum. Interact. 30, 4, Article 65 (sep 2023), 27 pages. https://doi.org/10.1145/3589952Google ScholarDigital Library
- Hao-Chiang Koong Lin, Yu-Hsuan Lin, Tao-Hua Wang, Lun-Ke Su, and Yueh-Min Huang. 2021. Effects of Incorporating Augmented Reality into a Board Game for High School Students’ Learning Motivation and Acceptance in Health Education. Sustainability 13, 6 (2021), 3333. https://doi.org/10.3390/su13063333Google ScholarCross Ref
- Jean-Luc Lugrin, Maximilian Ertl, Philipp Krop, Richard Klüpfel, Sebastian Stierstorfer, Bianka Weisz, Maximilian Rück, Johann Schmitt, Nina Schmidt, and Marc Erich Latoschik. 2018. Any “Body” There? Avatar Visibility Effects in a Virtual Reality Game. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Reutlingen, Germany, 17–24. https://doi.org/10.1109/VR.2018.8446229Google ScholarCross Ref
- Anabela Marto and Alexandrino Gonçalves. 2022. Augmented Reality Games and Presence: A Systematic Review. Journal of Imaging 2022, Vol. 8, Page 91 8 (3 2022), 91. Issue 4. https://doi.org/10.3390/JIMAGING8040091Google ScholarCross Ref
- Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino. 1995. Augmented reality: a class of displays on the reality-virtuality continuum. In Telemanipulator and Telepresence Technologies, Hari Das (Ed.). Vol. 2351. International Society for Optics and Photonics, SPIE, Boston, MA, United States, 282 – 292. https://doi.org/10.1117/12.197321Google ScholarCross Ref
- Shohei Mori, Sei Ikeda, and Hideo Saito. 2017. A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects. IPSJ Transactions on Computer Vision and Applications 9, 1 (2017), 1–14.Google ScholarCross Ref
- Noradila Nordin, Nur Rasyidah Mohd Nordin, and Wafa Omar. 2022. The Efficacy of REV-OPOLY Augmented Reality Board Game in Higher Education. International Journal of Emerging Technologies in Learning (iJET) 17, 07 (Apr. 2022), pp. 22–37. https://doi.org/10.3991/ijet.v17i07.26317Google ScholarCross Ref
- Nahal Norouzi, Gerd Bruder, Brandon Belna, Stefanie Mutter, Damla Turgut, and Greg Welch. 2019. A Systematic Review of the Convergence of Augmented Reality, Intelligent Virtual Agents, and the Internet of Things. Springer International Publishing, Cham, 1–24. https://doi.org/10.1007/978-3-030-04110-6_1Google ScholarCross Ref
- Heather L. O’Brien and Elaine G. Toms. 2008. What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology 59, 6 (2008), 938–955. https://doi.org/10.1002/asi.20801Google ScholarCross Ref
- Catherine S. Oh, Jeremy N. Bailenson, and Gregory F. Welch. 2018. A Systematic Review of Social Presence: Definition, Antecedents, and Implications. Frontiers in Robotics and AI 5 (Oct. 2018), 114. https://doi.org/10.3389/frobt.2018.00114Google ScholarCross Ref
- Heather L. O’Brien, Paul Cairns, and Mark Hall. 2018. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. International Journal of Human-Computer Studies 112 (April 2018), 28–39. https://doi.org/10.1016/j.ijhcs.2018.01.004Google ScholarCross Ref
- Alexander Plopski, Ada Virginia Taylor, Elizabeth Jeanne Carter, and Henny Admoni. 2019. InvisibleRobot: Facilitating Robot Manipulation Through Diminished Reality. In 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, Beijing, China, 165–166. https://doi.org/10.1109/ismar-adjunct.2019.00-55Google ScholarCross Ref
- Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, Andrew Y Ng, 2009. ROS: an open-source Robot Operating System. In ICRA workshop on open source software, Vol. 3. IEEE, Kobe, Japan, 5.Google Scholar
- Daniel Roth, Jean Luc Lugrin, Dmitri Galakhov, Arvid Hofmann, Gary Bente, Marc Erich Latoschik, and Arnulph Fuhrmann. 2016. Avatar realism and social interaction quality in virtual reality. Proceedings - IEEE Virtual Reality 2016-July (2016), 277–278. https://doi.org/10.1109/VR.2016.7504761 Avatar realism has a significant impact.Google ScholarCross Ref
- Shunsuke Saito, Gabriel Schwartz, Tomas Simon, Junxuan Li, and Giljoo Nam. 2023. Relightable Gaussian Codec Avatars. arxiv:2312.03704 [cs.GR]Google Scholar
- Martijn J Schuemie, Peter Van Der Straaten, Merel Krijn, and Charles APG Van Der Mast. 2001. Research on presence in virtual reality: A survey. Cyberpsychology & behavior 4, 2 (2001), 183–201.Google Scholar
- Mel Slater, Martin Usoh, and Anthony Steed. 1994. Depth of Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments 3, 2 (05 1994), 130–144. https://doi.org/10.1162/pres.1994.3.2.130Google ScholarDigital Library
- Vinicius Souza, Anderson Maciel, Luciana Nedel, and Regis Kopper. 2021. Measuring Presence in Virtual Environments: A Survey. ACM Comput. Surv. 54, 8, Article 163 (oct 2021), 37 pages. https://doi.org/10.1145/3466817Google ScholarDigital Library
- Patrick L. Strandholt, Oana A. Dogaru, Niels C. Nilsson, Rolf Nordahl, and Stefania Serafin. 2020. Knock on Wood: Combining Redirected Touching and Physical Props for Tool-Based Interaction in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (, Honolulu, HI, USA,) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376303Google ScholarDigital Library
- Ryo Suzuki, Adnan Karim, Tian Xia, Hooman Hedayati, and Nicolai Marquardt. 2022. Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (, New Orleans, LA, USA,) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 553, 33 pages. https://doi.org/10.1145/3491102.3517719Google ScholarDigital Library
- Kien TP Tran, Sungchul Jung, Simon Hoermann, and Robert W Lindeman. 2019. MDI: A Multi-channel Dynamic Immersion Headset for Seamless Switching between Virtual and Real World Activities. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, IEEE Press, Osaka, Japan, 350–358. https://doi.org/10.1109/VR.2019.8798240Google ScholarCross Ref
- Martin Usoh, Ernest Catena, Sima Arman, and Mel Slater. 2000. Using Presence Questionnaires in Reality. Presence: Teleoperators and Virtual Environments 9 (10 2000), 497–503. Issue 5. https://doi.org/10.1162/105474600566989Google ScholarDigital Library
- Raphael Vallat. 2018. Pingouin: statistics in Python. Journal of Open Source Software 3, 31 (2018), 1026. https://doi.org/10.21105/joss.01026Google ScholarCross Ref
- Michael Walker, Thao Phung, Tathagata Chakraborti, Tom Williams, and Daniel Szafir. 2023. Virtual, Augmented, and Mixed Reality for Human-robot Interaction: A Survey and Virtual Design Element Taxonomy. J. Hum.-Robot Interact. 12, 4, Article 43 (jul 2023), 39 pages. https://doi.org/10.1145/3597623Google ScholarDigital Library
- Tzu-Yang Wang, Yuji Sato, Mai Otsuki, Hideaki Kuzuoka, and Yusuke Suzuki. 2020. Effect of Body Representation Level of an Avatar on Quality of AR-Based Remote Instruction. Multimodal Technologies and Interaction 4, 1 (Feb. 2020), 3. https://doi.org/10.3390/mti4010003Google ScholarCross Ref
- Florian Weidner, Gerd Boettcher, Stephanie Arevalo Arboleda, Chenyao Diao, Luljeta Sinani, Christian Kunert, Christoph Gerhardt, Wolfgang Broll, and Alexander Raake. 2023. A Systematic Review on the Visualization of Avatars and Agents in AR & VR displayed using Head-Mounted Displays., 2596-2606 pages. Issue 5. https://doi.org/10.1109/TVCG.2023.3247072Google ScholarDigital Library
- Bob G. Witmer and Michael J. Singer. 1998. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoper. Virtual Environ. 7, 3 (jun 1998), 225–240. https://doi.org/10.1162/105474698565686Google ScholarDigital Library
- Yuelang Xu, Benwang Chen, Zhe Li, Hongwen Zhang, Lizhen Wang, Zerong Zheng, and Yebin Liu. 2023. Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians. arxiv:2312.03029 [cs.CV]Google Scholar
- Cik Suhaimi Yusof, Tian Sheng Low, Ajune Wanis Ismail, and Mohd Shahrizal Sunar. 2019. Collaborative Augmented Reality for Chess Game in Handheld Devices. In 2019 IEEE Conference on Graphics and Media (GAME). IEEE, Pulau Pinang, Malaysia, 32–37. https://doi.org/10.1109/GAME47560.2019.8980979Google ScholarCross Ref
- Andre Zenner and Antonio Krüger. 2017. Shifty: A weight-shifting dynamic passive haptic proxy to enhance object perception in virtual reality. IEEE transactions on visualization and computer graphics 23, 4 (2017), 1285–1294.Google ScholarDigital Library
- Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. 2023. Drivable 3D Gaussian Avatars. arxiv:2311.08581 [cs.CV]Google Scholar
Index Terms
- Exploring the Kuroko Paradigm: The Effect of Enhancing Virtual Humans with Reality Actuators in Augmented Reality
Recommendations
Augmented reality technologies, systems and applications
This paper surveys the current state-of-the-art of technology, systems and applications in Augmented Reality. It describes work performed by many different research groups, the purpose behind each new Augmented Reality system, and the difficulties and ...
Building and Employing Cross-Reality
This issue's Works in Progress department lists five projects with a focus on cross reality. The first two involve using Project Wonderland for collaborative projects. The next three use virtual and augmented reality to enhance user capabilities and ...
Extending Virtual Reality Display Wall Environments Using Augmented Reality
SUI '19: Symposium on Spatial User InteractionTwo major form factors for virtual reality are head-mounted displays and large display environments such as CAVE®and the LCD-based successor CAVE2®. Each of these has distinct advantages and limitations based on how they’re used. This work explores ...
Comments