skip to main content
10.1145/3570945.3607358acmconferencesArticle/Chapter ViewAbstractPublication PagesivaConference Proceedingsconference-collections
extended-abstract

The Affective Bar Piano

Published: 22 December 2023 Publication History

Abstract

Music is a great way of supporting a story. It adds a new layer of affective information and as such substantially increases the listening experience in storytelling scenarios. However, in real-time settings, creating emotionally fitting music requires permanent adaptation to the story's mood. While methods to compose and modify music according to emotional states are widely explored, current research rarely uses those techniques in a real-time setting, where such accompanying background music still requires improvisation by human musicians. In this work, we introduce the Affective Bar Piano, a virtual agent that assesses the mood of a story in real time. At the same time, the agent adapts its play to mirror the sensed affect of a human storyteller. In the presented demonstration scenario, the virtual agent is embodied by a 3D piano character playing music in a Wild West saloon setting.

References

[1]
Claus-Christian Carbon. 2018. Empirical aesthetics: In quest of a clear terminology and valid methodology. Exploring transdisciplinarity in art and sciences (2018), 107--119.
[2]
John D Carnwath and Alan S Brown. 2014. Understanding the value and impacts of cultural experiences. Manchester, United Kingdom: Arts Council England (2014).
[3]
Adyasha Dash and Kat R Agres. 2023. AI-Based Affective Music Generation Systems: A Review of Methods, and Challenges. arXiv preprint arXiv:2301.06890 (2023).
[4]
Akrivi Katifori, Sara Perry, Maria Vayanou, Angeliki Antoniou, Ioannis-Panagiotis Ioannidis, Sierra McKinney, Angeliki Chrysanthi, and Yannis Ioannidis. 2020. "Let them talk!" exploring guided group interaction in digital storytelling experiences. Journal on Computing and Cultural Heritage (JOCCH) 13, 3 (2020), 1--30.
[5]
Bettina Laugwitz, Theo Held, and Martin Schrepp. 2008. Construction and evaluation of a user experience questionnaire. In HCI and Usability for Education and Work: 4th Symposium of the Workgroup Human-Computer Interaction and Usability Engineering of the Austrian Computer Society, USAB 2008, Graz, Austria, November 20-21, 2008. Proceedings 4. Springer, 63--76.
[6]
Jerrold Levinson. 2004. Music as narrative and music as drama. Mind & Language 19, 4(2004), 428--441.
[7]
Florian Lingenfelser, Johannes Wagner, Jun Deng, Raymond Brueckner, Björn Schuller, and Elisabeth André. 2016. Asynchronous and event-based fusion systems for affect recognition on naturalistic data in comparison to conventional approaches. IEEE Transactions on Affective Computing 9, 4 (2016), 410--423.
[8]
Vera Micznik. 2001. Music and narrative revisited: Degrees of narrativity in Beethoven and Mahler. Journal of the Royal Musical Association 126, 2 (2001), 193--249.
[9]
Claudia Muth and Claus-Christian Carbon. 2013. The Aesthetic Aha: On the pleasure of having insights into Gestalt. Acta psychologica 144, 1 (2013), 25--30.
[10]
Hannes Ritschel, Ilhan Aslan, Silvan Mertes, Andreas Seiderer, and Elisabeth André. 2019. Personalized Synthesis of Intentional and Emotional Non-Verbal Sounds for Social Robots. In 8th International Conference on Affective Computing and Intelligent Interaction, ACII 2019, Cambridge, United Kingdom, September 3-6, 2019. IEEE, 1--7. https://doi.org/10.1109/ACII.2019.8925487
[11]
Ruben Schlagowski, Silvan Mertes, and Elisabeth André. 2021. Taming the chaos: exploring graphical input vector manipulation user interfaces for GANs in a musical context. In Proceedings of the 16th International Audio Mostly Conference. 216--223.
[12]
Ruben Schlagowski, Dariia Nazarenko, Yekta Can, Kunal Gupta, Silvan Mertes, Mark Billinghurst, and Elisabeth André. 2023. Wish You Were Here: Mental and Physiological Effects of Remote Music Collaboration in Mixed Reality. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1--16.
[13]
Marco Scirea, Julian Togelius, Peter Eklund, and Sebastian Risi. 2017. Affective evolutionary music composition with MetaCompose. Genetic Programming and Evolvable Machines 18 (2017), 433--465.
[14]
Narayanan Srinivasan. 2007. Cognitive neuroscience of creativity: EEG based approaches. Methods 42, 1 (2007), 109--116.
[15]
Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, and Björn W Schuller. 2023. Dawn of the transformer era in speech emotion recognition: closing the valence gap. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
[16]
Duncan Williams, Alexis Kirke, Eduardo Miranda, Ian Daly, Faustina Hwang, James Weaver, and Slawomir Nasuto. 2017. Affective calibration of musical feature sets in an emotionally intelligent music composition system. ACM Transactions on Applied Perception (TAP) 14, 3 (2017), 1--13.
[17]
Naomi Ziv and Maya Goshen. 2006. The effect of 'sad'and 'happy'background music on the interpretation of a story in 5 to 6-year-old children. British Journal of Music Education 23, 3 (2006), 303--314.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
IVA '23: Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents
September 2023
376 pages
ISBN:9781450399944
DOI:10.1145/3570945
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 December 2023

Check for updates

Author Tags

  1. affective computing
  2. interactive music
  3. musical storytelling
  4. virtual agents

Qualifiers

  • Extended-abstract
  • Research
  • Refereed limited

Conference

IVA '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 53 of 196 submissions, 27%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 51
    Total Downloads
  • Downloads (Last 12 months)41
  • Downloads (Last 6 weeks)4
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media