TelEmoScatter: Enabling Remote Interaction and Emotional Connections in Virtual and Physical Music Performance
Authors: 
Chen-Wei Fu, Wei-Lun Huang, Pin-Xuan Liu, Yu-Hsuan Chen, Ming-Cong Su, Andrew Chen, Ping-Hsuan Han, Tse-Yu PanAuthors Info & Claims
Chen-Wei Fu, Wei-Lun Huang, Pin-Xuan Liu, Yu-Hsuan Chen, Ming-Cong Su, Andrew Chen, Ping-Hsuan Han, Tse-Yu PanAuthors Info & ClaimsArticle No.: 103, Pages 1 - 3
Abstract
To enrich the emotional experiences of virtual reality (VR) online audiences in music performances, we developed TelEmoScatter, a system that facilitates remote interaction between music performers and onsite audiences. Our system also fosters emotional connections for online audiences through sound-visualization conversion, which is influenced by the state of the onsite audiences using computer vision techniques. In this work, we generate a 3D space using real-time sound-visualization techniques by converting MIDI signals from musical instruments into dynamic animations. Additionally, we employ video analysis to predict the emotions of the onsite audience, allowing seamless integration of emotional visual cues into the virtual scene. With our system, users can effortlessly immerse themselves in the emotional expressions of performers through music and experience the unique atmosphere of a live performance venue simply by wearing a VR headset.
References
[1]
Matthew N Bain. 2008. Real time music visualization: A study in the visual extension of music. Ph. D. Dissertation. The Ohio State University.
[2]
Chin-Han Chen, Ming-Fang Weng, Shyh-Kang Jeng, and Yung-Yu Chuang. 2008. Emotion-based music visualization using photos. In Advances in Multimedia Modeling: 14th International Multimedia Modeling Conference, MMM 2008, Kyoto, Japan, January 9-11, 2008. Proceedings 14. Springer, 358–368.
[3]
Lung-Pan Cheng, Sebastian Marwecki, and Patrick Baudisch. 2017. Mutual human actuation. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 797–805.
[4]
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arxiv:1704.04861 [cs.CV]
[5]
Glenn Jocher, Ayush Chaurasia, Alex Stoken, Jirka Borovec, NanoCode012, Yonghye Kwon, Kalen Michael, TaoXie, Jiacong Fang, imyhxy, Lorna, Zeng Yifu, Colin Wong, Abhiram V, Diego Montes, Zhiqiang Wang, Cristi Fati, Jebastin Nadar, Laughing, UnglvKitDe, Victor Sonck, tkianai, yxNONG, Piotr Skalski, Adam Hogan, Dhruv Nair, Max Strobel, and Mrinal Jain. 2022. ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation. https://doi.org/10.5281/zenodo.7347926
[6]
Arto Lehtiniemi and Jukka Holm. 2012. Using animated mood pictures in music recommendation. In 2012 16th International Conference on Information Visualisation. IEEE, 143–150.
[7]
Hugo B Lima, Carlos GR Dos Santos, and Bianchi S Meiguins. 2021. A survey of music visualization techniques. ACM Computing Surveys (CSUR) 54, 7 (2021), 1–29.
[8]
Purva Tendulkar, Abhishek Das, Aniruddha Kembhavi, and Devi Parikh. 2020. Feel the music: Automatically generating a dance for an input song. arXiv preprint arXiv:2006.11905 (2020).
[9]
Tai-Chen Tsai, Tse-Yu Pan, Min-Chun Hu, and Ya-Lun Tao. 2022. StimulusLoop: Game-Actuated Mutuality Artwork for Evoking Affective State. In Proceedings of the 30th ACM International Conference on Multimedia. 7245–7247.
[10]
Monu Verma, Santosh Kumar Vipparthi, and Girdhari Singh. 2021. AffectiveNet: Affective-Motion Feature Learning for Microexpression Recognition. IEEE MultiMedia 28, 1 (2021), 17–27. https://doi.org/10.1109/MMUL.2020.3021659
Index Terms
- TelEmoScatter: Enabling Remote Interaction and Emotional Connections in Virtual and Physical Music Performance
Recommendations
DualStage: Enhancing Emotional Connections through Music Visualization for Synchronizing Live and Virtual Performances
UbiComp '24: Companion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous ComputingThe pandemic has accelerated the adoption of virtual concerts, bridging the physical and virtual worlds for performers and audiences. However, the emotional connection of live concerts remains irreplaceable. To address this issue, we propose DualStage, a ...
Liquid Hands: Evoking Emotional States via Augmented Reality Music Visualizations
IMX '21: Proceedings of the 2021 ACM International Conference on Interactive Media ExperiencesMusic performances have transformed in unprecedented ways with the advent of digital music. Plenty of music visualizers enhance live performances in various forms, including LED display boards and holographic illustrations. However, the ...
Emotion-based music visualization using photos
MMM'08: Proceedings of the 14th international conference on Advances in multimedia modelingMusic players for personal computers are often featured with music visualization by generating animated patterns according to the music's low-level features such as loudness and spectrum. This paper proposes an emotion-based music player which ...
Comments
Information & Contributors
Information
Published In
December 2023
745 pages
ISBN:9798400702051
DOI:10.1145/3595916
Copyright © 2023 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 January 2024
Check for updates
Author Tags
Qualifiers
- Demonstration
- Research
- Refereed limited
Funding Sources
- The Audio-Visual Synergy and Multi-Screen Interactive Teaching Alliance
- National Science and Technology Council
Conference
MMAsia '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 59 of 204 submissions, 29%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 85Total Downloads
- Downloads (Last 12 months)54
- Downloads (Last 6 weeks)5
Reflects downloads up to 28 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format