skip to main content
10.1145/3687272.3688295acmconferencesArticle/Chapter ViewAbstractPublication PageshaiConference Proceedingsconference-collections
research-article
Open access

Virtual Reality “Space Agent” as a Socially Interactive Design Partner: A study investigating users’ perception of and interaction with Socially Interactive Virtual Environment (SIVE) as a design partner

Published: 24 November 2024 Publication History

Abstract

“Socially Interactive Virtual Environment” (SIVE) refers to a reconfigurable virtual environment that is perceived and interacted with as a socially intelligent agent. SIVE is a subset of “Space Agents”, which refers to intelligent or interactive agents that are both environmental (i.e. capable of reconfiguring virtual or physical spaces/environments) and human-like in (at least some of) their behavior. In this paper, we first proposed the concept of SIVE, and then explored the intricate human-SIVE interactions in the context of design partnership through empirical studies. More specifically, through the empirical studies, we aim to gain a preliminary understanding of how users perceive SIVE's social and environmental communications when SIVE is trying to play the role of a design partner, and how such user perceptions may shape the human-SIVE design partnership. Harnessing Unreal Engine, we designed and developed a virtual environment that used various visual and auditory modalities to interact with users and employed the Wizard-of-Oz (WoZ) technique to conduct in-lab experiments with 10 participants. The participants' perspectives and interactions were video recorded, and then carefully reviewed and analyzed by multiple researchers. A semi-structured interview was also conducted with each participant after the WoZ experiment, with the transcripts analyzed through the grounded theory coding technique. The findings of this study contribute to a preliminary understanding of the potential benefits and challenges associated with integrating SIVE into collaborative design practices and have the potential to shape the future of human- “intelligent agent” design partnerships and co-creations, offering guidance for designers, developers, and researchers seeking to leverage the social expressiveness of a virtual space agent as a design partner.

Supplemental Material

MP4 File
This is a video showing the preliminary prototype of SIVE as a design partner.

References

[1]
Mohammad Hossein Jarrahi, Christoph Lutz, and Gemma Newlands. 2022. Artificial intelligence, human intelligence and hybrid intelligence based on mutual augmentation. Big Data & Society 9, 2 (2022), 205395172211428. https://doi.org/10.1177/20539517221142824
[2]
Mohammad Hossein Jarrahi, David Askay, Ali Eshraghi, Preston Smith. 2023. Artificial intelligence and knowledge management: A partnership between human and AI. Business Horizons, Volume 66, Issue 1 (2023), Pages 87-99. https://doi.org/10.1016/j.bushor.2022.03.002
[3]
Julia Ostheimer, Soumitra Chowdhury, Sarfraz Iqbal. 2021. An alliance of humans and machines for machine learning: Hybrid intelligent systems and their design principles. Technology in Society, Volume 66 (2021), 101647. https://doi.org/10.1016/j.techsoc.2021.101647
[4]
Nan-Ning Zheng, Zi-Yi Liu, Peng-Ju Ren, Yong-Qiang Ma, Shi-Tao Chen, Si-Yu Yu, Jian-Ru Xue, Ba-Dong Chen, and Fei-Yue Wang. 2017. Hybrid-augmented intelligence: collaboration and cognition. Frontiers of Information Technology & Electronic Engineering 18, 2 (2017), 153–179. https://doi.org/10.1631/fitee.1700053
[5]
Zhuohao Wu, Danwen Ji, Kaiwen Yu, Xianxu Zeng, Dingming Wu, and Mohammad Shidujaman. 2021. AI Creativity and the Human-AI Co-creation Model. http://dx.doi.org/10.1007/978-3-030-78462-1_13
[6]
Sara Salevati Feldman. 2017. Co-Creation: Human and AI Collaboration in Creative Expression. http://dx.doi.org/10.14236/ewic/EVA2017.84
[7]
Lei Zhang and Steve Oney. 2020. FlowMatic: An Immersive Authoring Tool for Creating Interactive Scenes in Virtual Reality. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST '20). Association for Computing Machinery, New York, NY, USA, 342–353. https://doi.org/10.1145/3379337.3415824
[8]
Flavien Lécuyer, Valérie Gouranton, Adrien Reuzeau, Ronan Gaugne, and Bruno Arnaldi. 2020. Action Sequencing in VR, a No-Code Approach. In: Gavrilova, M., Tan, C., Chang, J., Thalmann, N. (eds) Transactions on Computational Science XXXVII. Lecture Notes in Computer Science, vol 12230. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-61983-4_4
[9]
Hugo Coelho, Pedro Monteiro, Guilherme Gonçalves, Miguel Melo, and Maximino Bessa. 2022. Authoring tools for virtual reality experiences: a systematic review. Multimedia Tools Appl. 81, 19 (Aug 2022), 28037–28060. https://doi.org/10.1007/s11042-022-12829-9
[10]
Steve Harrison and Paul Dourish. 1996. Re-place-ing space: the roles of place and space in collaborative systems. In Proceedings of the 1996 ACM conference on Computer supported cooperative work (CSCW '96). Association for Computing Machinery, New York, NY, USA, 67–76. https://doi.org/10.1145/240080.240193
[11]
Joshua McVeigh-Schultz, Anya Kolesnichenko, and Katherine Isbister. 2019. Shaping Pro-Social Interaction in VR: An Emerging Design Framework. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Paper 564, 1–12. https://doi.org/10.1145/3290605.3300794
[12]
Horst Robin, Naraghi-Taghi-Off Ramtin, Rau Linda, Doerner Ralf. 2022. Authoring With Virtual Reality Nuggets—Lessons Learned. Frontiers in Virtual Reality, Volume 3 (2022). https://doi.org/10.3389/frvir.2022.840729
[13]
Marc Steen. 2013. Co-Design as a Process of Joint Inquiry and Imagination. Design Issues. 29. 16-28. http://dx.doi.org/10.1162/DESI_a_00207
[14]
Kleinsmann, Maaike & Valkenburg, Rianne. 2008. Barriers and enablers for creating shared understanding in co-design projects. Design Studies. 29. 369-386. http://dx.doi.org/10.1016/j.destud.2008.03.003
[15]
Huybrechts, Liesbeth & Benesch, Henric & Geib, Jonathan. 2017. Institutioning: Participatory Design, Co-Design and the public realm. CoDesign. 13. 148-159. http://dx.doi.org/10.1080/15710882.2017.1355006
[16]
Peter Bradwell and Sarah Marr. making the most of collaboration an international survey of public service co-design DEMOS REPORT 23 in association with PwC's Public Sector Research Centre. Retrieved from https://demos.co.uk/wp-content/uploads/files/CollabWeb.pdf
[17]
Ryan, Marie-Laure. 1999. “Immersion vs. Interactivity: Virtual Reality and Literary Theory.” SubStance 28, no. 2, 110–37. https://doi.org/10.2307/3685793
[18]
Mel Slater. 2009. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B. 364, 1535 (Dec. 2009), 3549–3557. https://doi.org/10.1098/rstb.2009.0138
[19]
Greenwald S.W., Wiley Corning, and Pattie Maes. 2017. Multi-User Framework for Collaboration and Co-Creation in Virtual Reality. In Proceedings of the 12th International Conference on Computer Supported Collaborative Learning (CSCL).
[20]
Tomás Dorta, Gokce Kinayoglu, and Michael Hoffmann. 2016. Hyve-3D and the 3D Cursor: Architectural co-design with freedom in Virtual Reality. International Journal of Architectural Computing, 14(2), 87-102. https://doi.org/10.1177/1478077116638921
[21]
Hans-Christian Jetter, Roman Rädle, Tiare Feuchtner, Christoph Anthes, Judith Friedl, and Clemens Nylandsted Klokmose. 2020. "In VR, everything is possible!": Sketching and Simulating Spatially-Aware Interactive Spaces in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3313831.3376652
[22]
Anna Bilyatdinova, Andrey Karsakov, Alexey Bezgodov, and Alexey Dukhanov. 2016. Virtual environment for creative and collaborative learning. Adv. Intelligent Syst. Comput. 416, 371–381. http://dx.doi.org/10.1007/978-3-319-27478-2_26
[23]
Jorge Ferreira Franco, Sandra Regina da Cruz, and Roseli de Deus Lopes. 2006. Computer graphics, interactive technologies and collaborative learning synergy supporting individuals' skills development. In ACM SIGGRAPH 2006 Educators program (SIGGRAPH '06). Association for Computing Machinery, New York, NY, USA, 42–es. https://doi.org/10.1145/1179295.1179338
[24]
Larysa Nadolny, Joan Woolfrey, Matthew Pierlott, and Seth Kahn. 2013. SciEthics Interactive: science and ethics learning in a virtual environment. Educational technology research and development 61, 6 (2013), 979–999. https://doi.org/10.1007/s11423-013-9319-0
[25]
Palitha Edirisingha, Ming Nie, Mark Pluciennik, and Ruth Young. 2009. Socialisation for learning at a distance in a 3-D multi-user virtual environment. British Journal of Educational Technology 40, 3 (2009), 458–479. https://doi.org/10.1111/j.1467-8535.2009.00962.x
[26]
Fengfeng Ke and Sungwoong Lee. 2016. Virtual reality based collaborative design by children with high-functioning autism: design-based flexibility, identity, and norm construction. Interactive Learning Environments 24, 7 (2016), 1511–1533. https://doi.org/10.1080/10494820.2015.1040421
[27]
David Sirkin, Brian Mok, Stephen Yang, and Wendy Ju. 2015. Mechanical Ottoman: How Robotic Furniture Offers and Withdraws Support. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI '15). Association for Computing Machinery, New York, NY, USA, 11–18. https://doi.org/10.1145/2696454.2696461
[28]
Guy Hoffman and Wendy Ju. 2014. Designing robots with movement in mind. J. Hum.-Robot Interact. 3, 1 (February 2014), 91–122. https://doi.org/10.5898/JHRI.3.1.Hoffman
[29]
Nguyen, Alex Binh Vinh Duc & Simeone, Adalberto & Vande Moere, Andrew. 2021. Exploring an Architectural Framework for Human-Building Interaction via a Semi-Immersive Cross-Reality Methodology. http://dx.doi.org/10.1145/3434073.3444643
[30]
Yixiao Wang & Keith Evan Green. 2019. A Pattern-Based, Design Framework for Designing Collaborative Environments. In Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '19). Association for Computing Machinery, New York, NY, USA, 595–604. https://doi.org/10.1145/3294109.3295652
[31]
Yixiao Wang and Keith Evan Green. 2021. “Space Agency”: A “Strong Concept” for Designing Socially Interactive, Robotic Environments. In Lecture Notes in Computer Science. Lecture Notes in Computer Science, 295–307. https://doi.org/10.1007/978-3-030-90525-5_25
[32]
Yixiao Wang, Francois Guimbretiere, and Keith Evan Green. 2020. Are Space-making Robots, Agents? Investigations on User Perception of an Embedded Robotic Surface. https://doi.org/10.1109/ro-man47096.2020.9223532
[33]
Carlos Ramos, Juan Carlos Augusto, and Daniel Shapiro. 2008. Ambient Intelligence—the Next Step for Artificial Intelligence. IEEE Intelligent Systems 23, 2 (2008), 15–18. https://doi.org/10.1109/mis.2008.19
[34]
Michael Luck & Ruth Aylett. 2000. Applying artificial intelligence to virtual reality: Intelligent virtual environments. Applied Artificial Intelligence 14, 3–32. https://doi.org/10.1080/088395100117142
[35]
Harry T. Reis and Ladd Wheeler. 1991. Studying social interaction with the Rochester Interaction Record. Advances in Experimental Social Psychology, 24, 269–318. https://doi.org/10.1016/S0065-2601(08)60332-9
[36]
Libretexts. (2021, February 20). 5.1A: Understanding social interaction. Social Sci LibreTexts. https://socialsci.libretexts.org/Bookshelves/Sociology/Introduction_to_Sociology/Sociology_(Boundless)/05%3A_Social_Interaction/5.01%3A_Understanding_Social_Interaction/5.1A%3A_Understanding_Social_Interaction
[37]
Henna Lahti, Pirita Seitamaa-Hakkarainen, and Kai Hakkarainen. Collaboration patterns in computer-supported collaborative designing. Des. Stud. 2004;25:351–371. https://doi.org/10.1016/j.destud.2003.12.001
[38]
Laura Petri. “Concept analysis of interdisciplinary collaboration,” Nursing Forum, vol. 45, no. 2, pp. 73–82, 2010. https://doi.org/10.1111/j.1744-6198.2010.00167.x
[39]
Neal Dreamson. Online collaboration in design education: An experiment in real-time manipulation of prototypes and communication. Int. J. Art Des. Educ. 2017;36:188–199. https://doi.org/10.1111/jade.12079
[40]
Marc Steen, Menno Manschot, and Nicole De Koning. 2011. “Benefits of Co-Design in Service Design Projects.” International Journal of Design 5 (2): 53–60.
[41]
Woojung Chang and Steven A. Taylor. 2016. “The Effectiveness of Customer Participation in New Product Development: A Meta-Analysis.” Journal of Marketing 80 (1): 47–64. https://doi.org/10.1509/jm.14.0057
[42]
Jakob Trischler, Simon J. Pervan, Stephen J. Kelly, and Don R. Scott. 2018. “The Value of Codesign: The Effect of Customer Involvement in Service Design Teams.” Journal of Service Research 21 (1): 75–100. https://doi.org/10.1177/1094670517714060
[43]
Stuart A. Cockbill, Andrew May, and Val Mitchell. 2019. The assessment of meaningful outcomes from co-design: A case study from the energy sector. She Ji: The Journal of Design, Economics, and Innovation, 5(3), 188-208. https://doi.org/10.1016/j.sheji.2019.07.004
[44]
Peng Wang, Xiaoliang Bai, Mark Billinghurst, Shusheng Zhang, Xiangyu Zhang, Shuxia Wang, Weiping He, Yuxiang Yan, and Hongyu Ji. 2021. AR/MR Remote Collaboration on Physical Tasks: A Review, Robotics and Computer-Integrated Manufacturing, Volume 72, 2021, 102071, ISSN 0736-5845, https://doi.org/10.1016/j.rcim.2020.102071.
[45]
Seungwon Kim, Mark Billinghurst, and Kangsoo Kim. 2020. Multimodal interfaces and communication cues for remote collaboration. J Multimodal User Interfaces 14, 313–319 (2020). https://doi.org/10.1007/s12193-020-00346-8
[46]
Weidong Huang, Seungwon Kim, Mark Billinghurst, and Leila Alem. 2019. Sharing hand gesture and sketch cues in remote collaboration. J Vis Commun Image Represent 58:428–438. https://doi.org/10.1016/j.jvcir.2018.12.010
[47]
Seungwon Kim, Mark Billinghurst, and Gun Lee. 2018. The effect of collaboration styles and view independence on video-mediated remote collaboration. Comput Support Cooper Work 27(3):569–607. https://doi.org/10.1007/s10606-018-9324-2
[48]
Ronald M. Baecker, ed. 1994. Readings in groupware and computer-supported cooperative work: assisting human–human collaboration, 1st edn. Morgan Kaufmann, San Francisco.
[49]
Anthony DeVincenzi, Lining Yao, Hiroshi Ishii, and Ramesh Raskar. 2011. Kinected conference: augmenting video imaging with calibrated depth and audio. In Proceedings of the ACM 2011 conference on Computer supported cooperative work (CSCW '11). Association for Computing Machinery, New York, NY, USA, 621–624. https://doi.org/10.1145/1958824.1958929
[50]
Andrew Irlitti, Thammathip Piumsomboon, Daniel Jackson, and Bruce H. Thomas. 2019. Conveying spatial awareness cues in xR collaborations. IEEE Trans Visu Comput Graph 25(11):3178–3189. https://doi.org/10.1109/TVCG.2019.2932173
[51]
Geonsun Lee, HyeongYeop Kang, JongMin Lee, and JungHyun Han. 2020. A user study on view-sharing techniques for one-to-many mixed reality collaborations. In: Proceedings of the IEEE conference on virtual reality and 3D user interfaces, pp 343–352. https://doi.org/10.1109/VR46266.2020.00054
[52]
Simeon Taylor, Thuong Hoang, George Aranda, Gerard T. Mulvany, and Stefan Greuter. 2022. Immersive Collaborative VR Application Design. International Journal of Gaming and Computer-Mediated Simulations 13, 1–14. https://doi.org/10.4018/ijgcms.291538
[53]
Robert E. Kraut, Darren Gergle, and Susan R. Fussell. 2002. The use of visual information in shared visual spaces: informing the development of virtual co-presence. In Proceedings of the 2002 ACM conference on Computer supported cooperative work (CSCW '02). Association for Computing Machinery, New York, NY, USA, 31–40. https://doi.org/10.1145/587078.587084
[54]
Theophilus Teo, Louise Lawrence, Gun A. Lee, Mark Billinghurst, and Matt Adcock. 2019. Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Paper 201, 1–14. https://doi.org/10.1145/3290605.3300431
[55]
Monsurat Olaosebikan, Claudia Aranda Barrios, Blessing Kolawole, Lenore Cowen, and Orit Shaer. 2022. Identifying Cognitive and Creative Support Needs for Remote Scientific Collaboration using VR: Practices, Affordances, and Design Implications. In Proceedings of the 14th Conference on Creativity and Cognition (C&C '22). Association for Computing Machinery, New York, NY, USA, 97–110. https://doi.org/10.1145/3527927.3532797
[56]
Steven Dow, Jaemin Lee, Christopher Oezbek, 2005. Wizard of Oz interfaces for mixed reality applications. In: CHI ’05 Extended Abstracts on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, CHI EA ’05, p 1339–1342, https://doi.org/10.1145/1056808.1056911
[57]
Jakob Nielsen. 1993. Usability Engineering. https://doi.org/10.1016/c2009-0-21512-1
[58]
Jordan, Brigitte & Henderson, Austin. 1995. Interaction Analysis: Foundations and Practice. The Journal of the Learning Sciences. 4. 39-103. http://dx.doi.org/10.1207/s15327809jls0401_2
[59]
Kathy Charmaz. 2014. Constructing Grounded Theory (2nd. ed.). Sage, London, UK.
[60]
Wynn, David & Eckert, Claudia. 2017. Perspectives on iteration in design and development. Research in Engineering Design. 28. https://link.springer.com/article/10.1007/s00163-016-0226-3
[61]
Maier, Anja & Störrle, Harald. 2011. What are the Characteristics of Engineering Desing Processes?. ICED 11 - 18th International Conference on Engineering Design - Impacting Society Through Engineering Design. 1. 188-198.
[62]
Hoang Nam Le. 2012. A transformation-based model integration framework to support iteration management in engineering design. Ph.D. Dissertation. University of Cambridge, England, UK. https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.608020
[63]
Robert P. Smith and Steven D. Eppinger. 1997. Identifying Controlling Features of Engineering Design Iteration. Management Science 43, 3 (1997), 276–293. https://doi.org/10.1287/mnsc.43.3.276
[64]
Marc Steen, M. Manschot, and Nicole De Koning. 2011. Benefits of co-design in service design projects. International Journal of Design, 5(2), 53-60.
[65]
C. Hendricks Brown, Sheppard G. Kellam, Sheila Kaupert, Bengt O. Muthén, Wei Wang, Linda K. Muthén, Patricia Chamberlain, Craig L. Povey, Rick Cady, Thomas W. Valente, Mitsunori Ogihara, Guillermo J. Prado, Hilda M. Pantin, Carlos G. Gallo, José Szapocznik, Sara J. Czaja, and John W. Mcmanus. 2012. Partnerships for the Design, Conduct, and Analysis of Effectiveness, and Implementation Research: Experiences of the Prevention Science and Methodology Group. Administration and Policy in Mental Health and Mental Health Services Research 39, 4 (2012), 301–316. https://doi.org/10.1007/s10488-011-0387-3
[66]
Kathryn E. Ringland, LouAnne Boyd, Heather Faucett, Amanda L.L. Cullen, and Gillian R. Hayes. 2017. Making in Minecraft: A Means of Self-Expression for Youth with Autism. In Proceedings of the 2017 Conference on Interaction Design and Children (IDC '17). Association for Computing Machinery, New York, NY, USA, 340–345. https://doi.org/10.1145/3078072.3079749
[67]
Benjamin C. Riordan and Damian Scarf. 2017. Crafting minds and communities with Minecraft. F1000Research 5, (2017), 2339. https://doi.org/10.12688/f1000research.9625.2
[68]
Maria Cipollone, Catherine Schifter, and Rick Moffat. 2014. Minecraft as a Creative Tool: A Case Study. International Journal of Game-Based Learning. 4(2), 1-14. http://dx.doi.org/10.4018/ijgbl.2014040101
[69]
Katie Salen Tekinbaş, Krithika Jagannath, Ulrik Lyngs, and Petr Slovák. 2021. Designing for Youth-Centered Moderation and Community Governance in Minecraft. ACM Transactions on Computer-Human Interaction 28, 4 (2021), 1–41. https://doi.org/10.1145/3450290

Index Terms

  1. Virtual Reality “Space Agent” as a Socially Interactive Design Partner: A study investigating users’ perception of and interaction with Socially Interactive Virtual Environment (SIVE) as a design partner

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        HAI '24: Proceedings of the 12th International Conference on Human-Agent Interaction
        November 2024
        502 pages
        ISBN:9798400711787
        DOI:10.1145/3687272
        This work is licensed under a Creative Commons Attribution International 4.0 License.

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 24 November 2024

        Check for updates

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Funding Sources

        Conference

        HAI '24
        Sponsor:
        HAI '24: International Conference on Human-Agent Interaction
        November 24 - 27, 2024
        Swansea, United Kingdom

        Acceptance Rates

        Overall Acceptance Rate 121 of 404 submissions, 30%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • 0
          Total Citations
        • 94
          Total Downloads
        • Downloads (Last 12 months)94
        • Downloads (Last 6 weeks)47
        Reflects downloads up to 21 Jan 2025

        Other Metrics

        Citations

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Login options

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media