Skip to main content
SpringerLink
Log in

Inclusive Digital Storytelling: Artificial Intelligence and Augmented Reality to Re-centre Stories from the Margins

  • Conference paper
  • First Online:
Interactive Storytelling (ICIDS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14383))

Included in the following conference series:

  • 674 Accesses

Abstract

As the concept of the Metaverse becomes a reality, storytelling tools sharpen their teeth to include Artificial Intelligence and Augmented Reality as prominent enabling features. While digitally savvy and privileged populations are well-positioned to use technology, marginalized groups risk being left behind and excluded from societal progress, deepening the digital divide. In this paper, we describe MEMEX, an interactive digital storytelling tool where Artificial Intelligence and Augmented Reality play enabling roles in support of the cultural integration of communities at risk of exclusion. The tool was developed in the context of 3 years EU-funded project, and in this paper, we focus on describing its final working prototype with its pilot study.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://developers.google.com/ar.

  2. 2.

    https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home.

  3. 3.

    Bauhaus of the Seas Sails - https://cordis.europa.eu/project/id/101079995 and DCitizens - https://cordis.europa.eu/project/id/101079116.

References

  1. Aliprantis, J., Caridakis, G.: A survey of augmented reality applications in cultural heritage. Int. J. Comput. Methods Herit. Sci. (IJCMHS) 3(2), 118–147 (2019)

    Article  Google Scholar 

  2. Bala, P., James, S., Del Bue, A., Nisi, V.: Writing with (digital) scissors: designing a text editing tool for assisted storytelling using crowd-generated content. In: Vosmeer, M., Holloway-Attaway, L. (eds.) ICIDS 2022. LNCS, vol. 13762, pp. 139–158. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-22298-6_9

    Chapter  Google Scholar 

  3. Barbara, J., Haahr, M.: Identification and IDNs in the metaverse: who would we like to be? In: Vosmeer, M., Holloway-Attaway, L. (eds.) ICIDS 2022. LNCS, vol. 13762, pp. 601–615. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-22298-6_39

    Chapter  Google Scholar 

  4. Barthes, R., Howard, R., Barthes, R.: S/Z, repr edn. Blackwell, Oxford (1998)

    Google Scholar 

  5. Bekele, M.K., Pierdicca, R., Frontoni, E., Malinverni, E.S., Gain, J.: A survey of augmented, virtual, and mixed reality for cultural heritage. J. Comput. Cult. Herit. (JOCCH) 11(2), 1–36 (2018)

    Article  Google Scholar 

  6. Benford, S., Giannachi, G., Koleva, B., Rodden, T.: From interaction to trajectories: designing coherent journeys through user experiences. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 709–718 (2009)

    Google Scholar 

  7. Bleecker, J., Knowlton, J.: Locative media: a brief bibliography and taxonomy of GPS-enabled locative media. Leonardo Electron. Almanac 14,3 (Locative Media Special Issue) (2006)

    Google Scholar 

  8. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3(2), 77–101 (2006). https://doi.org/10.1191/1478088706qp063oa, http://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa

  9. Bravo, F.A., Hurtado, J.A., González, E.: Using robots with storytelling and drama activities in science education. Educ. Sci. 11(7), 329 (2021)

    Article  Google Scholar 

  10. Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)

    Google Scholar 

  11. Bruner, J.: Life as narrative. Soc. Res. 54(1), 11–32 (1987). http://www.jstor.org/stable/40970444, publisher: The New School

  12. Calderwood, A., Qiu, V., Gero, K.I., Chilton, L.B.: How novelists use generative language models: an exploratory user study. In: HAI-GEN+ user2agent@ IUI (2020)

    Google Scholar 

  13. Cavazza, M., Charles, F., Mead, S.J.: Character-based interactive storytelling. IEEE Intell. Syst. 17(4), 17–24 (2002)

    Article  MATH  Google Scholar 

  14. Cavazza, M., Charles, F., Mead, S.J.: Interactive storytelling: from AI experiment to new media. In: Proceedings of the Second International Conference on Entertainment Computing, pp. 1–8 (2003)

    Google Scholar 

  15. Cesário, V., Petrelli, D., Nisi, V.: Teenage visitor experience: classification of behavioral dynamics in museums. In: Bernhaupt, R., et al. (eds.) CHI ’20: CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 25–30 April 2020, pp. 1–13. ACM (2020). https://doi.org/10.1145/3313831.3376334

  16. Cosley, D., Sosik, V.S., Schultz, J., Peesapati, S.T., Lee, S.: Experiences with designing tools for everyday reminiscing. Hum.-Comput. Interact. 27(1–2), 175–198 (2012)

    Google Scholar 

  17. Emerson, L.: Reading Writing Interfaces: From the Digital to the Bookbound, vol. 44. U of Minnesota Press, Minneapolis (2014)

    Google Scholar 

  18. Erxleben, F., Günther, M., Krötzsch, M., Mendez, J., Vrandečić, D.: Introducing Wikidata to the linked data web. In: Mika, P., et al. (eds.) ISWC 2014. LNCS, vol. 8796, pp. 50–65. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11964-9_4

    Chapter  Google Scholar 

  19. Farman, J.: Site-specificity, pervasive computing, and the reading interface. AoIR Sel. Papers Internet Res. 3(0) (2013). https://spir.aoir.org/ojs/index.php/spir/article/view/8857

  20. Ferreira, M.J., Paradeda, R.B., Oliveira, R., Nisi, V., Paiva, A.: Using storytelling to teach children biodiversity. In: Vosmeer, M., Holloway-Attaway, L. (eds.) ICIDS 2022. LNCS, vol. 13762, pp. 3–27. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-22298-6_1

    Chapter  Google Scholar 

  21. Floch, J., Jiang, S.: One place, many stories digital storytelling for cultural heritage discovery in the landscape. In: 2015 Digital Heritage, vol. 2, pp. 503–510. IEEE (2015)

    Google Scholar 

  22. Frohlich, D.M., et al.:: StoryBank: mobile digital storytelling in a development context. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA, pp. 1761–1770. ACM, April 2009. https://doi.org/10.1145/1518701.1518972, https://dl.acm.org/doi/10.1145/1518701.1518972

  23. Gero, K.I., Liu, V., Chilton, L.: Sparks: Inspiration for science writing using language models. In: Designing Interactive Systems Conference. DIS ’22, pp. 1002–1019. Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3532106.3533533

  24. Giaccardi, E., Speed, C., Redström, J., Ben Allouch, S., Shklovski, I., Smith, R.C.: AI and the conditions of design: towards a new set of design ideals. In: Proceedings of the Design Research Society, Bilbao, Spain, June 2022. https://doi.org/10.21606/drs.2022.1078, https://dl.designresearchsociety.org/drs-conference-papers/drs2022/editorials/27

  25. Grudin, J.: From Tool to Partner: The Evolution of Human-Computer Interaction. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-02218-0

    Book  Google Scholar 

  26. Haahr, M., Wiil, P.H.: A lens to the past: using site-specific augmented reality for historical interpretation. In: Söbke, H., Spangenberger, P., Göbel, S. (eds.) JCSG 2022. LNCS, vol. 13476, pp. 259–265. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-15325-9_19

    Chapter  Google Scholar 

  27. Hargood, C., Hunt, V., Weal, M.J., Millard, D.E.: Patterns of sculptural hypertext in location based narratives. In: Proceedings of the 27th ACM Conference on Hypertext and Social Media. HT ’16, pp. 61–70. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2914586.2914595

  28. Harrison, S., Dourish, P.: Re-Place-Ing space: the roles of place and space in collaborative systems. In: Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work. CSCW ’96, Boston, Massachusetts, USA, pp. 67–76. Association for Computing Machinery, New York, NY, USA (1996). https://doi.org/10.1145/240080.240193

  29. Hassenzahl, M., Burmester, M., Koller, F.: AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In: Szwillus, G., Ziegler, J. (eds.) Mensch & Computer 2003: Interaktion in Bewegung, vol. 57, pp. 187–196. Springer (2003). https://doi.org/10.1007/978-3-322-80058-9_19

  30. Honnibal, M., Montani, I.: spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing (2017, to appear)

    Google Scholar 

  31. van den Hoven, E., Eggen, J.: The design of a recollection supporting device: a study into triggering personal recollections. In: 10th International Conference on Human-Computer Interaction (HCI International 2003), pp. 1034–1038. Lawrence Erlbaum (2003)

    Google Scholar 

  32. Hwang, A.H.C.: Too late to be creative? AI-empowered tools in creative processes. In: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems. CHI EA ’22. Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3491101.3503549

  33. Kampa, A., Spierling, U.: Smart authoring for location-based augmented reality storytelling applications. In: INFORMATIK 2017 (2017)

    Google Scholar 

  34. Lam, K.Y., Hang Lee, L., Braud, T., Hui, P.: M2A: a framework for visualizing information from mobile web to mobile augmented reality. In: 2019 IEEE International Conference on Pervasive Computing and Communications (PerCom), pp. 1–10 (2019). https://doi.org/10.1109/PERCOM.2019.8767388

  35. Lambert, J., Hessler, H.B.: Digital Storytelling: Capturing Lives, Creating Community, 5th edn, revised and updated edn. Routledge, Taylor & Francis Group, New York (2018)

    Google Scholar 

  36. Lane, G.: Urban tapestries: wireless networking, public authoring and social knowledge. Pers. Ubiquit. Comput. 7, 169–175 (2003). https://doi.org/10.1007/s00779-003-0229-8

    Article  Google Scholar 

  37. Lee, M., Liang, P., Yang, Q.: Coauthor: designing a human-AI collaborative writing dataset for exploring language model capabilities. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. CHI ’22. Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3491102.3502030

  38. Li, J., Wang, Z.: An interactive augmented reality graph visualization for Chinese painters. Electronics 11(15) (2022). https://doi.org/10.3390/electronics11152367, https://www.mdpi.com/2079-9292/11/15/2367

  39. Liang, C.A., Munson, S.A., Kientz, J.A.: Embracing four tensions in human-computer interaction research with marginalized people. ACM Trans. Comput.-Hum. Interact. 28(2) (2021). https://doi.org/10.1145/3443686

  40. Magerko, B.: A proposal for an interactive drama architecture. In: AAAI 2002 Spring Symposium Series: Artificial Intelligence and Interactive Entertainment (2002)

    Google Scholar 

  41. Mateas, M.: An Oz-centric review of interactive drama and believable agents. In: Wooldridge, M.J., Veloso, M. (eds.) Artificial Intelligence Today. LNCS (LNAI), vol. 1600, pp. 297–328. Springer, Heidelberg (1999). https://doi.org/10.1007/3-540-48317-9_12

    Chapter  Google Scholar 

  42. McAdams, D.P.: The psychology of life stories. Rev. General Psychol. 5(2), 100–122 (2001). SAGE Publications, Sage CA: Los Angeles, CA

    Google Scholar 

  43. Mohamed, H.A., et al.: Geolocation of cultural heritage using multi-view knowledge graph embedding. arXiv preprint arXiv:2209.03638 (2022)

  44. Mohamed, H.A., Vascon, S., Pilutti, D., James, S.: Memex-kg (2021). https://github.com/MEMEXProject/MEMEX-KG

  45. Nicenboim, I., Giaccardi, E., Redström, J.: From explanations to shared understandings of AI. In: Proceedings of the Design Research Conference. Bilbao, Spain, June 2022. https://doi.org/10.21606/drs.2022.773, https://dl.designresearchsociety.org/drs-conference-papers/drs2022/researchpapers/293

  46. Nisi, V.: Location Aware Multimedia Stories: A Location Based View of Interactive Narrative. Lambert Academic Publishing, Saarbrücken (2011)

    Google Scholar 

  47. Nisi, V., Bala, P., Bostock, H., Cesário, V., Nunes, N.J.: “before gentrification, we claim for habitation”: Eliciting values and assets through cultural heritage storytelling. In: Proceedings of the 2023 ACM Designing Interactive Systems Conference. DIS ’23, pp. 2423–2436. Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3563657.3596124

  48. Nisi, V., Bala, P., Cesário, V., Stuart, J., Bue, A.D., Nunes, N.J.: “connected to the people”: Social Inclusion & Cohesion in Action Through a cultural heritage digital tool, vol. 7. Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3610168

  49. Nisi, V., Bostock, H., Cesário, V., Acedo, A., Nunes, N.: Impalpable narratives: how to capture intangible cultural heritage of migrant communities. In: Proceedings of the 10th International Conference on Communities and Technologies - Wicked Problems in the Age of Tech. C &T ’21, pp. 109–120. Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3461564.3461575

  50. Nisi, V., Costanza, E., Dionisio, M.: Placing location-based narratives in context through a narrator and visual markers. Interact. Comput. 29(3), 287–305 (2017). https://doi.org/10.1093/iwc/iww020

  51. Nisi, V., Dionisio, M., Hanna, J., Ferreira, L., Nunes, N.: Yasmine’s adventures: an interactive urban experience exploring the sociocultural potential of digital entertainment. In: Chorianopoulos, K., Divitini, M., Hauge, J.B., Jaccheri, L., Malaka, R. (eds.) ICEC 2015. LNCS, vol. 9353, pp. 343–356. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24589-8_26

    Chapter  Google Scholar 

  52. Nisi, V., Haahr, M.: Weird view: interactive multilinear narratives and real-life community stories, 2, 27 (2006)

    Google Scholar 

  53. Nisi, V., Oakley, I., Boer, M.d.: Locative narratives as experience: a new perspective on location aware multimedia stories. In: Proceedings of ArTech 2010, Porto, Portugal (2010)

    Google Scholar 

  54. Nisi, V., Oakley, I., Haahr, M.: Inner city locative media: design and experience of a location-aware mobile narrative for the Dublin liberties neighborhood. Intell. Agent 6(2) (2006)

    Google Scholar 

  55. Nisi, V., Wood, A., Davenport, G., Oakley, I.: Hopstory: an interactive, location-based narrative distributed in space and time. In: Göbel, S., et al. (eds.) TIDSE 2004. LNCS, vol. 3105, pp. 132–141. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-27797-2_18

    Chapter  Google Scholar 

  56. O’hara, K., Tuffield, M.M., Shadbolt, N.: Lifelogging: privacy and empowerment with memories for life. Identity Inf. Soc. 1, 155–172 (2008)

    Google Scholar 

  57. Perkis, A., et al.: INDCOR White Paper 2: Interactive Narrative Design for Representing Complexity (2023). _eprint: 2305.01925

    Google Scholar 

  58. R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2021). https://www.R-project.org/

  59. Reid, J., Hull, R., Cater, K., Fleuriot, C.: Magic moments in situated mediascapes. In: Proceedings of the 2005 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology. ACE ’05, pp. 290–293. Association for Computing Machinery, New York, NY, USA (2005). https://doi.org/10.1145/1178477.1178529

  60. Shin, J.e., Woo, W.: Design guidelines for a location-based digital heritage storytelling tool to support author intent. In: 2018 3rd Digital Heritage International Congress (DigitalHERITAGE) held jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018), pp. 1–8. IEEE (2018)

    Google Scholar 

  61. Szilas, N.: Interactive drama on computer: beyond linear narrative. In: AAAI Fall Symposium on Narrative Intelligence, vol. 144, pp. 150–156 (1999)

    Google Scholar 

  62. Szilas, N.: Stepping into the interactive drama. In: Göbel, S., et al. (eds.) TIDSE 2004. LNCS, vol. 3105, pp. 14–25. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-27797-2_3

    Chapter  Google Scholar 

  63. Thue, D.: Working with intelligent narrative technologies. In: Hargood, C., Millard, D.E., Mitchell, A., Spierling, U. (eds.) The Authoring Problem. Human–Computer Interaction Series, pp. 271–284. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-05214-9_17

    Chapter  Google Scholar 

  64. Weizenbaum, J.: Eliza-a computer program for the study of natural language communication between man and machine. Commun. ACM 9(1), 36–45 (1966)

    Article  Google Scholar 

  65. Wickham, H.: ggplot2: Elegant Graphics for Data Analysis. Springer, New York (2009). https://doi.org/10.1007/978-0-387-98141-3, http://ggplot2.org

  66. Yates, F.A.: The Art of Memory. University of Chicago Press, Chicago (2001). https://press.uchicago.edu/ucp/books/book/chicago/A/bo91674300.html

  67. Young, R.M.: Creating interactive narrative structures: the potential for AI approaches. Psychology 13, 1–26 (2000)

    Google Scholar 

  68. Zhang, Q., Hinze, A., Vanderschantz, N.: Narrative navigation: visualizing story order and locations in augmented reality. In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, pp. 541–542 (2020)

    Google Scholar 

  69. Zhang, Z., et al.: Storybuddy: a human-AI collaborative chatbot for parent-child interactive storytelling with flexible parental involvement. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–21 (2022)

    Google Scholar 

Download references

Acknowledgements

This research was supported by MEMEX (MEMories and EXperiences for Inclusive Digital Storytelling) Horizon Europe under GA 870743, by LARSyS (Project UIDB/50009/2020), and by the Portuguese Recovery and Resilience Program (PRR), IAPMEI/ANI/FCT under Agenda no.26, C645022399-00000057 (eGamesLab).

Specifically, we acknowledge the partners of MEMEX, including EY Advisory SpA (EY), NOHO Limited (Noho), University Ca’Foscari di Venezia (UNIVE), for their support in technological developments and documentation. As well as Michael Culture Association (MCA), Fundacio Interarts per a la Cooperacio Cultural Internacional (IA), Mapa das Ideias (MDI), Dédale (DED), and European Centre for Cultural Organisation & Management (ECCOM) for their support in evaluation. Full team members are available on the MEMEX project website (https://memexproject.eu/en/partners/consortium).

Author information

Authors and Affiliations

  1. ITI/LARSyS, IST-U., Lisbon, Portugal

    Valentina Nisi, Paulo Bala & Nuno Jardim Nunes

  2. Pattern Analysis and Computer Vision (PAVIS), Fondazione Istituto Italiano di Tecnologia (IIT), Genoa, Italy

    Stuart James & Alessio Del Bue

Authors
  1. Valentina Nisi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stuart James
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paulo Bala
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alessio Del Bue
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nuno Jardim Nunes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Valentina Nisi .

Editor information

Editors and Affiliations

  1. University of Skövde, Skövde, Sweden

    Lissa Holloway-Attaway

  2. University of Central Florida, Orlando, FL, USA

    John T. Murray

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nisi, V., James, S., Bala, P., Del Bue, A., Nunes, N.J. (2023). Inclusive Digital Storytelling: Artificial Intelligence and Augmented Reality to Re-centre Stories from the Margins. In: Holloway-Attaway, L., Murray, J.T. (eds) Interactive Storytelling. ICIDS 2023. Lecture Notes in Computer Science, vol 14383. Springer, Cham. https://doi.org/10.1007/978-3-031-47655-6_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-47655-6_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-47654-9

  • Online ISBN: 978-3-031-47655-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation