Abstract
The purpose of this paper is to study the correlation between attention and affective response when a user interacts with the artistic installation MovieScape - an Extended Immersive Digital Music Instrument. We analyzed the affective modulation in ten subjects. For that, we applied Affective Slider and audio-segmentation on the recorded performances of a Circumplex Space (a 2D Circumplex Model of Affect) to apply quantitative metrics and divide the interaction in Explorative and Contemplative states. In our paper we propose this as a unifying conceptual framework for analysing affective behavior within interactive installation such as MS. Overall, the analyses present an increase of Arousal after the interaction of the subjects within the environment, and a correlation between the Explorative interaction and the increment of Arousal which confirms the improvement in attention. Eventually, an inductive mechanism of expectancy occurred. These conclusions open up for potential in designing EIDMIs for therapeutic and pedagogical purposes. Also, these findings may also prove that subjects reached Creative Empowerment while becoming aware of MovieScape.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Abbreviations
The following abbreviations are used in this manuscript: AS: Affective Sliders, CE: Creative Empowerment, DMI: Digital Musical Instruments, EIDMI: Extended Immersive Digital Music Instruments, MS: MovieScape.
References
Abel, R. (ed.): Encyclopedia of Early Cinema. Routledge, Abingdon (2010)
Abel, R., Altman, R.: The Sounds of Early Cinema (1st ed.). Indiana University Press (2001). https://doi.org/10.1353/not.2002.0123
Altman, R.: Silent Film Sound. Columbia University Press, New York (2004)
Anderson, M.L.: Embodied cognition: a field guide. Artif. Intell. 149(1), 91–130 (2003). https://doi.org/10.1016/S0004-3702(03)00054-7
Berger, D.S.: Music therapy, sensory integration and the autistic child (5. impr, Vol. 22). Jessica Kingsley (2002)
Betella, A., Verschure, P.F.M.J.: The affective slider: a digital self-assessment scale for the measurement of human emotions. PLoS ONE 11(2), 11 (2016). https://doi.org/10.1371/journal.pone.0148037
Bullock, J.: Libxtract: a lightweight library for audio feature extraction. In: Proceedings of the 2007 International Computer Music Conference, no. 4 (2007)
Burch, N.: Life to Those Shadows. University of California Press, Oakland (1990)
Çamcı, A., Hamilton, R.: Audio-first VR: new perspectives on musical experiences in virtual environments. J. New Music Res. 49(1), 1–7 (2020). https://doi.org/10.1080/09298215.2019.1707234
Cannam, C., Landone, C., Sandler, M.: Sonic visualiser: an open source application for viewing, analysing, and annotating music audio files. In: Proceedings of the 18th ACM International Conference on Multimedia, p. 1467 (2010). https://doi.org/10.1145/1873951.1874248
Châteauvert, J., Gaudreault, A.: The noises of spectators, or the spectator as additive to the spectacle. In: Abel, R., Altman, R. (Eds.), The Sounds of Early Cinema (1st ed., pp. 183–191). Indiana University Press (2001). https://doi.org/10.1353/not.2002.0123
Correa, A.G.D., Ficheman, I.K., do Nascimento, M., de Deus Lopes, R.: Computer assisted music therapy: a case study of an augmented reality musical system for children with cerebral palsy rehabilitation. In: 2009 Ninth IEEE International Conference on Advanced Learning Technologies, pp. 218–220 (2009). https://doi.org/10.1109/ICALT.2009.111
Etkin, A., Büchel, C., Gross, J.J.: The neural bases of emotion regulation. Nat. Rev. Neurosci. 16(11), 693–700 (2015). https://doi.org/10.1038/nrn4044
Caruana, F., Borghi, A.M.: Embodied Cognition: Una nuova psicologia. G. Italiano Di Psicol. 1, 23–48 (2013). https://doi.org/10.1421/73973
Gaudreault, A.: Narration and monstration in the cinema. J. Film Video Univ. Illinois Press 39(2), 29–36. JSTOR (1987). https://www.jstor.org/stable/20687768
Gaudreault, A., Dulac, N., Hidalgo, S. (eds.): A Companion to Early Cinema (1st ed.). Wiley-Blackwell, Hoboken (2012)
Gorbman, C.: Unheard Melodies: Narrative Film Music (1st ed.). BFI Pub - Indiana University Press, Bloomington (1987)
Gunning, T.: The cinema of attractions: early cinema, its spectator, and the avant-garde. Wide Angle 8(3–4), 63–70 (1986)
Gunning, T.: An aesthetic of astonishment: early film and the incredulous spectator. In: Williams, L. (Ed.), Viewing positions: Ways of seeing film (Reprint from 1989 Art&Text, nos. 34, 1st, pp. 114–133). Rutgers University Press (1995)
Hakvoort, L., de Jong, S., van de Ree, M., Kok, T., Macfarlane, C., de Haan, H.: Music therapy to regulate arousal and attention in patients with substance use disorder and posttraumatic stress disorder: a feasibility study. J. Music Ther. 57(3), 353–378 (2020). https://doi.org/10.1093/jmt/thaa007
Hebling, E.D.: As coletâneas musicais temáticas e os manuais: Uma introdução e um estudo de caso sobre os temas do exoticismo no cinema mudo [Master, Universidade de Campinas UNICAMP] (2017). http://repositorio.unicamp.br/jspui/handle/REPOSIP/330905
Hebling, E.D., et al.: MovieScape: audiovisual landscapes for silent movie: enactive experience in a multimodal installation. In: Proceedings of the 9th International Conference on Digital and Interactive Arts, pp. 1–7 (2019). https://doi.org/10.1145/3359852.3359883
Huron, D.: Sweet Anticipation: Music and the Psychology of Expectation. MIT Press, Cambridge (2006). https://doi.org/10.1016/s0262-4079(17)31427-6
Jeon, M., Fiebrink, R., Edmonds, E.A., Herath, D.: From rituals to magic: interactive art and HCI of the past, present, and future. Int. J. Hum. Comput. Stud. 131, 108–119 (2019). https://doi.org/10.1016/j.ijhcs.2019.06.005
Kanehira, R., Ito, Y., Suzuki, M., Hideo, F.: Enhanced relaxation effect of music therapy with VR. In: 2018 14th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), pp. 1374–1378 (2018). https://doi.org/10.1109/FSKD.2018.8686951
Lee, D., Seo, H., Jung, M.W.: Neural basis of reinforcement learning and decision making. Annu. Rev. Neurosci. 35(1), 287–308 (2012). https://doi.org/10.1146/annurev-neuro-062111-150512
Manzolli, J.: Multimodal generative installations and the creation of new Art form based on interactivity narratives, pp. 32–45. In: XVIII Generative Art Conference GA2015 (2015). https://www.generativeart.com/
McClelland, T.: The mental affordance hypothesis. Mind 129(514), 401–427 (2020). https://doi.org/10.1093/mind/fzz036
Musser, C.: The nickelodeon era begins: establishing the framework for Hollywood’s mode of representation. Framework J. Cinema Med. 22/23, 4–11 (1983). https://doi.org/10.5040/9781838710170
Nam, H.Y., Nitsche, M.: Interactive installations as performance: Inspiration for HCI. In: Proceedings of the 8th International Conference on Tangible, Embedded and Embodied Interaction - TEI ‘14, pp. 189–196 (2013). https://doi.org/10.1145/2540930.2540976
Noroozi, F., Corneanu, C.A., Kamińska, D., Sapiński, T., Escalera, S., Anbarjafari, G.: Survey on Emotional Body Gesture Recognition (2018). ArXiv:1801.07481 [Cs]. http://arxiv.org/abs/1801.07481
O’Kelly, J., James, L., Palaniappan, R., Taborin, J., Fachner, J., Magee, W.L.: Neurophysiological and behavioral responses to music therapy in vegetative and minimally conscious states. Front. Hum. Neurosci. 7 (2013). https://doi.org/10.3389/fnhum.2013.00884
Partesotti, E., Peñalba, A., Manzolli, J.: Digital instruments and their uses in music therapy. Nord. J. Music. Ther. 27(5), 399–418 (2018). https://doi.org/10.1080/08098131.2018.1490919
Partesotti, E.: (forthcoming) Immersive Digital Music Instruments to Empower Wellbeing through Creativity in Creating Digitally. Shifting Boundaries: Arts and Technologies - contemporary applications and concepts (2023). Springer
Partridge, D., Rowe, J.: Computers and creativity. Intellect (1994)
Pelletier, C.L.: The effect of music on decreasing arousal due to stress: a meta-analysis. J. Music Ther. 41(3), 192–214 (2004). https://doi.org/10.1093/jmt/41.3.192
Peñalba, A., Valles, M.-J., Partesotti, E., Sevillano, M.-Á., Castañón, R.: Accessibility and participation in the use of an inclusive musical instrument: the case of MotionComposer. J. Music Technol. Educ. 12(1), 79–94 (2019). https://doi.org/10.1386/jmte.12.1.79_1
Posner, J., Russell, J.A., Peterson, B.S.: The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17(03) (2005). https://doi.org/10.1017/S0954579405050340
Ryan, K., Schiavio, A.: Extended musicking, extended mind, extended agency. Notes on the third wave. New Ideas Psychol. 55, 8–17 (2019). https://doi.org/10.1016/j.newideapsych.2019.03.001
Sauer, R.: Photoplay music: a reusable repertory for silent film scoring, 1914–1929. American Music Research Center - Colorado Boulder, pp. 55–76 (1998)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction (Second edition). The MIT Press, Cambridge (2018)
Tahiroğlu, K.: Ever-shifting roles in building, composing and performing with digital musical instruments. J. New Music Res. 50(2), 155–164 (2021). https://doi.org/10.1080/09298215.2021.1900275
Taneja, A., Vishal, S.B., Mahesh, V., Geethanjali, B.: Virtual reality based neuro-rehabilitation for mental stress reduction. In: 2017 Fourth International Conference on Signal Processing, Communication and Networking (ICSCN), pp. 1–5 (2017). https://doi.org/10.1109/ICSCN.2017.8085665
Ward, D., Stapleton, M.: Es are good: cognition as enacted, embodied, embedded, affective and extended. In: Paglieri, F. (Ed.), Advances in Consciousness Research, vol. 86, pp. 89–104. John Benjamins Publishing Company (2012). https://doi.org/10.1075/aicr.86.06war
Wassermann, K.C., Eng, K., Verschure, P.F.M.J., Manzolli, J.: Live soundscape composition based on synthetic emotions. IEEE Multimed. 10(4), 82–90 (2003). https://doi.org/10.1109/MMUL.2003.1237553
Zamecnik, J.S.: Sam Fox Moving Picture Music vol. 1. Sam Fox Publisher, Cleveland (1913a)
Zamecnik, J.S.: Sam Fox Moving Picture Music vol. 2. Sam Fox Publisher, Cleveland (1913b)
Zamecnik, J.S.: Sam Fox Moving Picture Music vol. 3. Sam Fox Publisher, Cleveland (1914)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Elena Partesotti is supported by FAPESP, under Grant n. 2016/226190; Eduardo D. Hebling is supported by CAPES, under Grant n. 88887.313062/2019-00; Jônatas Manzolli is supported by CNPq projects, under Grant n. 429620/2018-7, 304431/2018-4; Micael Antunes is supported by FAPESP, under Grant n. 2019/09734-3.
Rights and permissions
Copyright information
© 2023 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Partesotti, E. et al. (2023). Analysis of Affective Behavior in the Artistic Installation Moviescape. In: Brooks, A.L. (eds) ArtsIT, Interactivity and Game Creation. ArtsIT 2022. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 479. Springer, Cham. https://doi.org/10.1007/978-3-031-28993-4_23
Download citation
DOI: https://doi.org/10.1007/978-3-031-28993-4_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-28992-7
Online ISBN: 978-3-031-28993-4
eBook Packages: Computer ScienceComputer Science (R0)