Skip to main content
SpringerLink
Log in

Generative AI for Immersive Experiences: Integrating Text-to-Image Models in VR-Mediated Co-design Workflows

  • Conference paper
  • First Online:
HCI International 2023 Posters (HCII 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1836))

Included in the following conference series:

Abstract

Text-to-image AI models can generate novel images for design inspiration. Yet, their applications for collaborative design (co-design) purposes and interoperability within simulation-based, immersive settings have been scarcely explored. In this paper, we propose a novel, multi-modal approach for interactive public participation in urban design projects. The main objectives of our research are (a) to describe a methodological workflow of integrating text-to-image AI models into VR-mediated co-design workshops, and (b) to investigate the applicability of the proposed workflow through a set of completed and prospective case studies. Both studies are parts of a broader research project, which aims to revitalize the city of Derby, UK through producing a series of sustainable design visions. Through these case studies, we discuss some preliminary results and introduce our future work.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Portman, M.E., Natapov, A., Fisher-Gewirtzman, D.: To go where no man has gone before: virtual reality in architecture, landscape architecture and environmental planning. Comput. Environ. Urban Syst. 54, 376–384 (2015)

    Article  Google Scholar 

  2. Meenar, M., Kitson, J.: Using multi-sensory and multi-dimensional immersive virtual reality in participatory planning. Urban Sci. 4(3), 34 (2020)

    Article  Google Scholar 

  3. Zhang, C., Zeng, W., Liu, L.: UrbanVR: an immersive analytics system for context-aware urban design. Comput. Graph. 99, 128–138 (2021)

    Article  Google Scholar 

  4. Safikhani, S., Keller, S., Schweiger, G., Pirker, J.: Immersive virtual reality for extending the potential of building information modeling in architecture, engineering, and construction sector: systematic review. Int. J. Dig. Earth 15(1), 503–526 (2022)

    Article  Google Scholar 

  5. Liu, X.: Three-dimensional visualized urban landscape planning and design based on virtual reality technology. IEEE Access 8, 149510–149521 (2020)

    Article  Google Scholar 

  6. Schrom-Feiertag, H., Stubenschrott, M., Regal, G., Matyus, T., Seer, S.: An interactive and responsive virtual reality environment for participatory urban planning. In: Proceedings of the 11th Annual Symposium on Simulation for Architecture and Urban Design pp. 1–7 (2020)

    Google Scholar 

  7. Kim, S., Kim, J., Kim, B.: Immersive virtual reality-aided conjoint analysis of urban square preference by living environment. Sustainability 12(16), 6440 (2020)

    Article  Google Scholar 

  8. Yu, R., Gu, N., Lee, G., Khan, A.: A systematic review of architectural design collaboration in immersive virtual environments. Designs 6(5), 93 (2022)

    Article  Google Scholar 

  9. Panya, D.S., Kim, T., Choo, S.: An interactive design change methodology using a BIM-based Virtual Reality and Augmented Reality. J. Build. Eng. 106030 (2023)

    Google Scholar 

  10. Delgado, J.M.D., Oyedele, L., Demian, P., Beach, T.: A research agenda for augmented and virtual reality in architecture, engineering and construction. Adv. Eng. Inform. 45, 101122 (2020)

    Article  Google Scholar 

  11. Michels, A.: Citizen participation in local policy making: design and democracy. Int. J. Public Adm. 35(4), 285–292 (2012)

    Article  Google Scholar 

  12. Binder, T., Brandt, E.: The design: lab as platform in participatory design research. CoDesign 4(2), 115–129 (2008)

    Article  Google Scholar 

  13. Nabatchi, T., Leighninger, M.: Citizenship, outside the public square. In: Public Participation for 21st Century Democracy, pp. 1–12 (2015)

    Google Scholar 

  14. Sanders, E.B.-N., Stappers, P.J.: Co-creation and the new landscapes of design. CoDesign 4(1), 5–18 (2008)

    Article  Google Scholar 

  15. Pillai, A.G., et al.: Communicate, Critique and Co-create (CCC) future technologies through design fictions in VR environment. In: Companion Publication of the 2020 ACM Designing Interactive Systems Conference, pp. 413–416 (2020)

    Google Scholar 

  16. Paasch Knudsen, S., Husted Hansen, H., Ørngreen, R.: Exploring the Learning Potentials of Augmented Reality Through Speculative Design, pp. 156–163 (2022)

    Google Scholar 

  17. Gu, N., Amini Behbahani, P.: A critical review of computational creativity in built environment design. Buildings 11(1), 29 (2021)

    Article  Google Scholar 

  18. Yildirim, E.: Text-to-image generation - AI in architecture. Art Archit. Theory Pract. Exper. 97 (2022)

    Google Scholar 

  19. Kingma, D.P., Welling, M.: Auto-Encoding Variational Bayes. CoRR, abs/1312.6114 (2013)

    Google Scholar 

  20. Goodfellow, I., et al.: Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020)

    Article  MathSciNet  Google Scholar 

  21. Open AI: DALL·E 2 (2002). https://openai.com/product/dall-e-2

  22. Stability AI: Stable Diffusion Public Release (2023). https://stability.ai/blog/stable-diffusion-public-release

  23. Google Research: Imagen: Text-to-Image Diffusion Models (2022). https://imagen.research.google/

  24. Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., Taigman, Y.: Make-a-scene: scene-based text-to-image generation with human priors. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, Tal (eds.) Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV, pp. 89–106. Springer Nature Switzerland, Cham (2022). https://doi.org/10.1007/978-3-031-19784-0_6

    Chapter  Google Scholar 

  25. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2022-June, pp. 10674–10685 (2022)

    Google Scholar 

  26. Saharia, C., et al.: Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022)

    Google Scholar 

  27. Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022)

    Google Scholar 

  28. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Adv. Neural. Inf. Process. Syst. 33, 6840–6851 (2020)

    Google Scholar 

  29. Dhariwal, P., Nichol, A.: Diffusion models beat GANs on image synthesis. Adv. Neural. Inf. Process. Syst. 11, 8780–8794 (2021)

    Google Scholar 

  30. Gu, S., et al.: Vector quantized diffusion model for text-to-image synthesis. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2022-June, pp. 10686–10696 (2022)

    Google Scholar 

  31. Seneviratne, S., Senanayake, D., Rasnayaka, S., Vidanaarachchi, R., Thompson, J.: DALLE-URBAN: capturing the urban design expertise of large text to image transformers (2022)

    Google Scholar 

  32. UrbanistAI: UrbanistAI (2023). https://www.urbanistai.com/

  33. Ehab, A., Heath, T.: Exploring immersive co-design: comparing human interaction in real and virtual elevated urban spaces in london. Sustain. 15(12), 9184 (2023). https://doi.org/10.3390/su15129184

    Article  Google Scholar 

Download references

Funding

This project has been generously supported by the Osborne Legacy. The financial assistance provided by the legacy has been instrumental in the successful completion of this research effort.

Author information

Authors and Affiliations

  1. Derby Urban Sustainable Transition (DUST), University of Derby, Derby, UK

    Chris Bussell, Ahmed Ehab, Daniel Hartle-Ryan & Timo Kapsalis

Authors
  1. Chris Bussell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmed Ehab
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel Hartle-Ryan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Timo Kapsalis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Timo Kapsalis .

Editor information

Editors and Affiliations

  1. University of Crete and Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

  2. Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Margherita Antona

  3. Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Stavroula Ntoa

  4. University of Central Florida, Orlando, FL, USA

    Gavriel Salvendy

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bussell, C., Ehab, A., Hartle-Ryan, D., Kapsalis, T. (2023). Generative AI for Immersive Experiences: Integrating Text-to-Image Models in VR-Mediated Co-design Workflows. In: Stephanidis, C., Antona, M., Ntoa, S., Salvendy, G. (eds) HCI International 2023 Posters. HCII 2023. Communications in Computer and Information Science, vol 1836. Springer, Cham. https://doi.org/10.1007/978-3-031-36004-6_52

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-36004-6_52

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-36003-9

  • Online ISBN: 978-3-031-36004-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation