G S Rajshekar Reddy
ABSTRACT
Music performances have transformed in unprecedented ways with the advent of digital music. Plenty of music visualizers enhance live performances in various forms, including LED display boards and holographic illustrations. However, the impracticability of live performances due to the CoVID-19 outbreak has led to event organizers adopting alternatives in virtual environments. In this work, we propose Liquid Hands, an Augmented Reality (AR) music visualizer system, wherein three-dimensional particles react to the flow of music, forming a visually aesthetic escapade. With hand-particle interactions, Liquid Hands aims to enrich the music listening experience in one’s personal space and bridge the gap between virtual and physical concerts. We intend to explore the emotions our system induces by conducting a pilot study, in which we measure the user’s psychological state through Electroencephalography (EEG). We hypothesize that the proposed system will evoke emotions akin to those exhibited in live music performances.
- Franz Aurenhammer and Rolf Klein. 2000. Voronoi Diagrams.Handbook of computational geometry 5, 10 (2000), 201–290.Google Scholar
- Ruth A Baer. 2011. Measuring mindfulness. Contemporary Buddhism 12, 1 (2011), 241–261.Google ScholarCross Ref
- Felicia Rodica Baltes and Andrei C Miu. 2014. Emotions during live music performance: Links with individual differences in empathy, visual imagery, and mood.Psychomusicology: Music, Mind, and Brain 24, 1 (2014), 58.Google Scholar
- Tony Bergstrom, Karrie Karahalios, and John C. Hart. 2007. Isochords: Visualizing Structure in Music. In Proceedings of Graphics Interface 2007 (Montreal, Canada) (GI ’07). Association for Computing Machinery, New York, NY, USA, 297–304. https://doi.org/10.1145/1268517.1268565Google ScholarDigital Library
- Jennifer Block-Lerner, Kristalyn Salters-Pedneault, and Matthew T Tull. 2005. Assessing mindfulness and experiential acceptance. In Acceptance and mindfulness-based approaches to anxiety. Springer, 71–99.Google Scholar
- The blog at the bottom of the sea. 2016. Fast Voronoi Diagrams and Distance Field Textures on the GPU With the Jump Flooding Algorithm. Retrieved March 27, 2021 from https://blog.demofox.org/2016/02/29/fast-voronoi-diagrams-and-distance-dield-textures-on-the-gpu-with-the-jump-flooding-algorithm/Google Scholar
- Kirk Warren Brown and Richard M Ryan. 2003. The benefits of being present: mindfulness and its role in psychological well-being.Journal of personality and social psychology 84, 4(2003), 822.Google Scholar
- Ross Buck. 1999. The biological affects: a typology.Psychological review 106, 2 (1999), 301.Google Scholar
- Elaine Chew and Alexandre R.J. Francois. 2003. MuSA.RT: Music on the Spiral Array. Real-Time. In Proceedings of the Eleventh ACM International Conference on Multimedia (Berkeley, CA, USA) (MULTIMEDIA ’03). Association for Computing Machinery, New York, NY, USA, 448–449. https://doi.org/10.1145/957013.957106Google ScholarDigital Library
- Peter Ciuha, Bojan Klemenc, and Franc Solina. 2010. Visualization of Concurrent Tones in Music with Colours. In Proceedings of the 18th ACM International Conference on Multimedia (Firenze, Italy) (MM ’10). Association for Computing Machinery, New York, NY, USA, 1677–1680. https://doi.org/10.1145/1873951.1874320Google ScholarDigital Library
- Mihaly Csikszentmihalyi and Mihaly Csikzentmihaly. 1990. Flow: The psychology of optimal experience. Vol. 1990. Harper & Row New York.Google Scholar
- deadmau5. 2021. Cubev3. Retrieved March 27, 2021 from https://cubev3.com/Google Scholar
- Frank M Diaz. 2013. Mindfulness, attention, and flow during music listening: An empirical investigation. Psychology of Music 41, 1 (2013), 42–58.Google ScholarCross Ref
- Simon Dixon. 2000. Extraction of musical performance parameters from audio data. In Proceedings of the First IEEE Pacific-Rim Conference on Multimedia. 42–45.Google Scholar
- Hauke Egermann, Mary Elizabeth Sutherland, Oliver Grewe, Frederik Nagel, Reinhard Kopiez, and Eckart Altenmüller. 2011. Does music listening in a social context alter experience? A physiological and psychological perspective on emotion. Musicae Scientiae 15, 3 (2011), 307–323.Google ScholarCross Ref
- Joyce Horn Fonteles, Maria Andréia Formico Rodrigues, and Victor Emanuel Dias Basso. 2013. Creating and evaluating a particle system for music visualization. Journal of Visual Languages & Computing 24, 6 (2013), 472–482.Google ScholarDigital Library
- Alf Gabrielsson. 2001. Emotions in strong experiences with music.Series in affective science. Music and emotion: Theory and research (2001), 431–449.Google Scholar
- Christian Gold, Martin Voracek, and Tony Wigram. 2004. Effects of music therapy for children and adolescents with psychopathology: a meta-analysis. Journal of Child Psychology and Psychiatry 45, 6 (2004), 1054–1063.Google ScholarCross Ref
- M. Goodwin and M. Vetterli. 1996. Time-frequency signal models for music analysis, transformation, and synthesis. In Proceedings of Third International Symposium on Time-Frequency and Time-Scale Analysis (TFTS-96). 133–136.Google Scholar
- Hewlett-Packard. 2021. HP VR Backpack. Retrieved March 27, 2021 from https://www8.hp.com/us/en/vr/vr-backpack.htmlGoogle Scholar
- R. Hiraga and N. Matsuda. 2004. Graphical expression of the mood of music. In 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763), Vol. 3. 2035–2038 Vol.3.Google Scholar
- Rumi Hiraga, Reiko Mizaki, and Issei Fujishiro. 2002. Performance Visualization: A New Challenge to Music through Visualization. In Proceedings of the Tenth ACM International Conference on Multimedia (Juan-les-Pins, France) (MULTIMEDIA ’02). Association for Computing Machinery, New York, NY, USA, 239–242. https://doi.org/10.1145/641007.641054Google ScholarDigital Library
- R. Hiraga, F. Watanabe, and I. Fujishiro. 2002. Music learning through visualization. In Second International Conference on Web Delivering of Music, 2002. WEDELMUSIC 2002. Proceedings.101–108.Google Scholar
- Carroll E Izard. 2007. Basic emotions, natural kinds, emotion schemas, and a new paradigm. Perspectives on psychological science 2, 3 (2007), 260–280.Google Scholar
- Patrik N Juslin, Simon Liljeström, Daniel Västfjäll, Gonçalo Barradas, and Ana Silva. 2008. An experience sampling study of emotional reactions to music: listener, music, and situation.Emotion 8, 5 (2008), 668.Google Scholar
- Patrik N Juslin and Marcel R Zentner. 2001. Current trends in the study of music and emotion: Overture. Musicae scientiae 5, 1_suppl (2001), 3–21.Google ScholarCross Ref
- Jon Kabat-Zinn and Jon Kabat Zinn. 2013. Mindfulness meditation in everyday life. BettterListen! LLC.Google Scholar
- Satoshi Kawase and Satoshi Obata. 2016. Psychological responses to recorded music as predictors of intentions to attend concerts: Emotions, liking, performance evaluations, and monetary value. Musicae Scientiae 20, 2 (2016), 163–172.Google ScholarCross Ref
- Carol L Krumhansl. 1997. An exploratory study of musical emotions and psychophysiology.Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale 51, 4 (1997), 336.Google Scholar
- Ondřej Kubelka. 2000. Interactive music visualization. In Central European Seminar on Computer Graphics, Vol. 4.Google Scholar
- You-Yun Lee and Shulan Hsieh. 2014. Classifying different emotional states by means of EEG-based functional connectivity patterns. PloS one 9, 4 (2014), e95415.Google ScholarCross Ref
- Lars-Olov Lundqvist, Fredrik Carlsson, Per Hilmersson, and Patrik N Juslin. 2009. Emotional responses to music: Experience, expression, and physiology. Psychology of music 37, 1 (2009), 61–90.Google Scholar
- Cathy H McKinney, Michael H Antoni, Mahendra Kumar, Frederick C Tims, and Philip M McCabe. 1997. Effects of guided imagery and music (GIM) therapy on mood and cortisol in healthy adults.Health psychology 16, 4 (1997), 390.Google Scholar
- Microsoft. 2020. Altspace VR. Retrieved March 27, 2021 from https://altvr.com/Google Scholar
- Microsoft. 2021. What is the Mixed Reality Toolkit. Retrieved March 27, 2021 from https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/Google Scholar
- Andrei C Miu and Felicia Rodica Balteş. 2012. Empathy manipulation impacts music-induced emotions: A psychophysiological study on opera. PloS one 7, 1 (2012), e30618.Google ScholarCross Ref
- Reiko Miyazaki and Issei Fujishiro. 2002. comp-i: 3d visualization of midi datasets. In VIS’02: Proceedings of the conference on Visualization 2002 Posters.Google Scholar
- Reiko Miyazaki, Issei Fujishiro, and Rumi Hiraga. 2003. Exploring MIDI Datasets. In ACM SIGGRAPH 2003 Sketches & Applications(San Diego, California) (SIGGRAPH ’03). Association for Computing Machinery, New York, NY, USA, 1. https://doi.org/10.1145/965400.965453Google ScholarDigital Library
- Ivan Nyklíček, Julian F Thayer, and Lorenz JP Van Doornen. 1997. Cardiorespiratory differentiation of musically-induced emotions.Journal of Psychophysiology(1997).Google Scholar
- OpenBCI. 2020. Introducing: Galea. Retrieved May 5, 2021 from https://openbci.com/community/introducing-galea-bci-hmd-biosensing/Google Scholar
- Jaak Panksepp. 2007. Neurologizing the psychology of affects: How appraisal-based constructivism and basic emotion theory can coexist. Perspectives on psychological science 2, 3 (2007), 281–296.Google Scholar
- Eric Prydz. 2021. Epic 5.0. Retrieved March 27, 2021 from https://www.ericprydz.com/epicGoogle Scholar
- Martin Roberts. 2018. The Unreasonable Effectiveness of Quasirandom Sequences. Retrieved March 27, 2021 from http://extremelearning.com.au/unreasonable-effectiveness-of-quasirandom-sequences/Google Scholar
- Damien Constantine Rompapas, Daniel Flores Quiros, Charlton Rodda, Bryan Christopher Brown, Noah Benjamin Zerkin, and Alvaro Cassinelli. 2020. Project Esky: Enabling High Fidelity Augmented Reality Content on an Open Source Platform. In ACM Interactive Surfaces and Spaces.Google Scholar
- Klaus R Scherer. 2004. Which emotions can be induced by music? What are the underlying mechanisms? And how can we measure them?Journal of new music research 33, 3 (2004), 239–251.Google Scholar
- Klaus R Scherer and Marcel R Zentner. 2001. Emotional effects of music: Production rules.(2001).Google Scholar
- S. M. Smith and G. N. Williams. 1997. A visualization of music. In Proceedings. Visualization ’97 (Cat. No. 97CB36155). 499–503.Google ScholarCross Ref
- Stereolabs. 2021. Zed Mini - Mixed Reality Camera. Retrieved March 27, 2021 from https://www.stereolabs.com/zed-mini/Google Scholar
- Aman Tiwari. 2018. MeshToSDF. Retrieved March 27, 2021 from https://github.com/aman-tiwari/MeshToSDFGoogle Scholar
- Tomorrowland. 2021. Around the World. Retrieved March 27, 2021 from https://www.tomorrowland.com/en/digital/around-the-world/welcomeGoogle Scholar
- Ultraleap. 2021. Leap Motion Controller. Retrieved March 27, 2021 from https://www.ultraleap.com/product/leap-motion-controller/Google Scholar
- Ultraleap. 2021. Project Northstar. Retrieved March 27, 2021 from https://developer.leapmotion.com/northstarGoogle Scholar
- Unity. 2021. Unity Home Page. Retrieved March 27, 2021 from https://unity.com/Google Scholar
- Unity. 2021. Visual Effect Graph. Retrieved March 27, 2021 from https://unity.com/visual-effect-graphGoogle Scholar
- L. Valbom and A. Marcos. 2007. An Immersive Musical Instrument Prototype. IEEE Computer Graphics and Applications 27, 4 (2007), 14–19.Google ScholarDigital Library
- M. Wattenberg. 2002. Arc diagrams: visualizing structure in strings. In IEEE Symposium on Information Visualization, 2002. INFOVIS 2002.110–116.Google ScholarCross Ref
- Wikipedia. 2020. Particle system. Retrieved March 27, 2021 from https://en.wikipedia.org/wiki/Particle_systemGoogle Scholar
- Wikipedia. 2021. Brownian motion. Retrieved March 27, 2021 from https://en.wikipedia.org/wiki/Brownian_motionGoogle Scholar
- Wikipedia. 2021. Signed distance function. Retrieved March 27, 2021 from https://en.wikipedia.org/wiki/Signed_distance_functionGoogle Scholar
- Robert Williams. 2020. Fortnite’ virtual rap concert draws record 12.3M attendees. Retrieved March 27, 2021 from https://www.mobilemarketer.com/news/fortnite-virtual-rap-concert-draws-record-123m-attendeesGoogle Scholar
- Marcel Zentner, Didier Grandjean, and Klaus R Scherer. 2008. Emotions evoked by the sound of music: characterization, classification, and measurement. Emotion 8, 4 (2008), 494.Google ScholarCross Ref
Recommendations
A Spatial Music Listening Experience in Augmented Reality
UIST '21 Adjunct: Adjunct Proceedings of the 34th Annual ACM Symposium on User Interface Software and TechnologyLive music provides a more immersive and social experience that recorded music cannot replicate. In a live music setting, listeners perceive sounds differently based on their position with respect to the musicians and can enjoy the experience with ...
An evaluation of augmented reality music notation
VRCAI '19: Proceedings of the 17th International Conference on Virtual-Reality Continuum and its Applications in IndustryWe conducted a focus group study of a prototype application to test the opportunities and limitations of augmented reality music notation for musical performance and rehearsal.
Multimodal augmented reality: the norm rather than the exception
MVAR '16: Proceedings of the 2016 workshop on Multimodal Virtual and Augmented RealityAugmented reality (AR) is commonly seen as a technology that overlays virtual imagery onto a participant's view of the world. In line with this, most AR research is focused on what we see. In this paper, we challenge this focus on vision and make a case ...
Comments