skip to main content
10.1145/3581783.3611968acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

GrooveMeter: Enabling Music Engagement-aware Apps by Detecting Reactions to Daily Music Listening via Earable Sensing

Published: 27 October 2023 Publication History

Abstract

We present GrooveMeter, a novel system that automatically detects vocal and motion reactions to music and supports music engagement-aware applications. We use smart earbuds as sensing devices, already widely used for music listening, and devise reaction detection techniques by leveraging an inertial measurement unit (IMU) and a microphone on earbuds. To explore reactions in daily music-listening situations, we collect the first-kind-of dataset containing 926-minute-long IMU and audio data with 30 participants. With the dataset, we discover unique challenges in detecting music-listening reactions and devise sophisticated processing pipelines to enable accurate and efficient detection. Our comprehensive evaluation shows GrooveMeter achieves the macro F1 scores of 0.89 for vocal reaction and 0.81 for motion reaction with leave-one-subject-out (LOSO) cross-validation (CV). More importantly, it shows higher accuracy and robustness compared to alternative methods. We also present the potential use cases.

Supplemental Material

MP4 File
In this video, we introduce GrooveMeter, a novel mobile system that automatically detects vocal and motion reactions while people listen to music in their daily lives through earable sensing. We first present potential applications that benefit from automatic music reaction detection. Detecting people's responses to music in various everyday situations using earable devices poses several challenges. We detail our proposed processing pipelines to address the challenges, which feature novel techniques such as early-stage filtering and leveraging musical information. We also show the results of the rigorous evaluation of GrooveMeter using datasets collected in noisy real-life situations where people typically enjoy music. The evaluation demonstrates the system's accuracy, robustness, and efficiency. We hope that viewers of this video will find valuable insights and ideas for its potential applications.

References

[1]
[n.d.]. Music Listening 2022. https://www.ifpi.org/wp-content/uploads/2022/11/ Engaging-with-Music-2022_full-report-1.pdf. Accessed: Apr. 22, 2023.
[2]
[n.d.]. TensorFlow Lite. https://www.tensorflow.org/lite. Accessed: Apr. 22, 2023.
[3]
Yudhik Agrawal, Samyak Jain, Emily Carlson, Petri Toiviainen, and Vinoo Alluri. 2020. Towards Multimodal MIR: Predicting individual differences from music-induced movement. arXiv preprint arXiv:2007.10695 (2020).
[4]
Xuan Bao, Songchun Fan, Alexander Varshavsky, Kevin Li, and Romit Roy Choudhury. 2013. Your reactions suggest you liked the movie: Automatic content rating via reaction sensing. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. 197--206.
[5]
Abdelkareem Bedri, Richard Li, Malcolm Haynes, Raj Prateek Kosaraju, Ishaan Grover, Temiloluwa Prioleau, Min Yan Beh, Mayank Goel, Thad Starner, and Gregory Abowd. 2017. EarBit: using wearable sensors to detect eating episodes in unconstrained environments. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 1, 3 (2017), 1--20.
[6]
Donald J Berndt and James Clifford. 1994. Using dynamic time warping to find patterns in time series. In Proceedings of the 3rd international conference on knowledge discovery and data mining. 359--370.
[7]
Anne J Blood and Robert J Zatorre. 2001. Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion. Proceedings of the national academy of sciences 98, 20 (2001), 11818--11823.
[8]
Birgitta Burger, Marc R Thompson, Geoff Luck, Suvi Saarikallio, and Petri Toivi-ainen. 2013. Influences of rhythm-and timbre-related musical features on characteristics of music-induced movement. Frontiers in psychology 4 (2013), 183.
[9]
Birgitta Burger, Marc R Thompson, Geoff Luck, Suvi H Saarikallio, and Petri Toiviainen. 2014. Hunting for the beat in the body: on period and phase locking in music-induced movement. Frontiers in human neuroscience 8 (2014), 903.
[10]
Gaoshuai Cao, Kuang Yuan, Jie Xiong, Panlong Yang, Yubo Yan, Hao Zhou, and Xiang-Yang Li. 2020. EarphoneTrack: involving earphones into the ecosystem of acoustic motion tracking. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems. 95--108.
[11]
Andrea Ferlini, Dong Ma, Robert Harle, and Cecilia Mascolo. 2021. EarGate: gait-based user identification with in-ear microphones. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 337--349.
[12]
Davide Figo, Pedro C Diniz, Diogo R Ferreira, and Joao MP Cardoso. 2010. Pre-processing techniques for context recognition from accelerometer data. Personal and Ubiquitous Computing 14, 7 (2010), 645--662.
[13]
G David Forney. 1973. The viterbi algorithm. Proc. IEEE 61, 3 (1973), 268--278.
[14]
Rolf Inge Godøy and Marc Leman. 2010. Musical gestures: Sound, movement, and meaning. Routledge.
[15]
Victor E Gonzalez-Sanchez, Agata Zelechowska, and Alexander Refsum Jensenius. 2018. Correspondences between music and involuntary human micromotion during standstill. Frontiers in psychology 9 (2018), 1382.
[16]
Susan Hallam, Ian Cross, and Michael Thaut. 2016. The Oxford Handbook of Music Psychology (2nd ed.). Oxford University Press.
[17]
Yan He, George Chernyshov, Jiawen Han, Dingding Zheng, Ragnar Thomsen, Danny Hynds, Muyu Liu, Yuehui Yang, Yulan Ju, Yun Suen Pai, et al. 2022. Frisson Waves: Exploring Automatic Detection, Triggering and Sharing of Aesthetic Chills in Music Performances. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1--23.
[18]
Romain Hennequin, Anis Khlif, Felix Voituret, and Manuel Moussallam. 2020. Spleeter: a fast and efficient music source separation tool with pre-trained models. Journal of Open Source Software 5, 50 (2020), 2154. https://doi.org/10.21105/joss. 02154 Deezer Research.
[19]
Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson. 2017. CNN Architectures for Large-Scale Audio Classification. In International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://arxiv.org/abs/1609.09430
[20]
Nan Jiang, Terence Sim, and Jun Han. 2022. EarWalk: towards walking posture identification using earables. In Proceedings of the 23rd Annual International Workshop on Mobile Computing Systems and Applications. 35--40.
[21]
Fahim Kawsar, Chulhong Min, Akhil Mathur, and Allesandro Montanari. 2018. Earables for personal-scale behavior analytics. IEEE Pervasive Computing 17, 3 (2018), 83--89.
[22]
Prerna Khanna, Tanmay Srivastava, Shijia Pan, Shubham Jain, and Phuc Nguyen. 2021. JawSense: recognizing unvoiced sound using a low-cost ear-worn system. In Proceedings of the 22nd International Workshop on Mobile Computing Systems and Applications. 44--49.
[23]
Rushil Khurana, Karan Ahuja, Zac Yu, Jennifer Mankoff, Chris Harrison, and Mayank Goel. 2018. GymCam: Detecting, recognizing and tracking simultaneous exercises in unconstrained scenes. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 4 (2018), 1--17.
[24]
Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. 2018. CREPE: A Convolutional Representation for Pitch Estimation. arXiv:1802.06182 [eess.AS]
[25]
Stefan Koelsch, Thomas Fritz, D Yves v. Cramon, Karsten Müller, and Angela D Friederici. 2006. Investigating emotion with music: an fMRI study. Human brain mapping 27, 3 (2006), 239--250.
[26]
Seungchul Lee, Chulhong Min, Alessandro Montanari, Akhil Mathur, Youngjae Chang, Junehwa Song, and Fahim Kawsar. 2019. Automatic Smile and Frown Recognition with Kinetic Earables. In Proceedings of the 10th Augmented Human International Conference 2019. 1--4.
[27]
Daniel J Levitin, Jessica A Grahn, and Justin London. 2018. The psychology of music: Rhythm and movement. Annual review of psychology 69 (2018), 51--75.
[28]
Geoff Luck, Suvi Saarikallio, Birgitta Burger, Marc R Thompson, and Petri Toivi-ainen. 2010. Effects of the Big Five and musical genre on music-induced movement. Journal of Research in Personality 44, 6 (2010), 714--720.
[29]
Emily Lynar, Erin Cvejic, Emery Schubert, and Ute Vollmer-Conna. 2017. The joy of heartfelt music: An examination of emotional and physiological responses. International Journal of Psychophysiology 120 (2017), 118--125.
[30]
Dong Ma, Andrea Ferlini, and Cecilia Mascolo. 2021. OESense: employing occlusion effect for in-ear human sensing. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services. 175--187.
[31]
Claudio Martella, Ekin Gedik, Laura Cabrera-Quiros, Gwenn Englebienne, and Hayley Hung. 2015. How was it? Exploiting smartphone sensing to measure implicit audience responses to live performances. In Proceedings of the 23rd ACM international conference on Multimedia. 201--210.
[32]
Chulhong Min, Akhil Mathur, and Fahim Kawsar. 2018. Exploring Audio and Kinetic Sensing on Earable Devices. In Proceedings of the 4th ACM Workshop on Wearable Systems and Applications (Munich, Germany) (WearSys '18). Association for Computing Machinery, New York, NY, USA, 5--10. https://doi.org/10.1145/ 3211960.3211970
[33]
Dan Morris, T Scott Saponas, Andrew Guillory, and Ilya Kelner. 2014. RecoFit: using a wearable sensor to find, recognize, and count repetitive exercises. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 3225--3234.
[34]
Shahriar Nirjon, Robert F Dickerson, Qiang Li, Philip Asare, John A Stankovic, Dezhi Hong, Ben Zhang, Xiaofan Jiang, Guobin Shen, and Feng Zhao. 2012. Musicalheart: A hearty way of listening to music. In Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems. 43--56.
[35]
Adrian C North, David J Hargreaves, and Jon J Hargreaves. 2004. Uses of music in everyday life. Music perception 22, 1 (2004), 41--77.
[36]
Francisco Javier Ordóñez and Daniel Roggen. 2016. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors 16, 1 (2016), 115.
[37]
Alisun Pawley and Daniel Müllensiefen. 2012. The science of singing along: A quantitative field study on sing-along behavior in the north of England. Music Perception 30, 2 (2012), 129--146.
[38]
Charissa Ann Ronao and Sung-Bae Cho. 2016. Human activity recognition with smartphone sensors using deep learning neural networks. Expert systems with applications 59 (2016), 235--244.
[39]
Burr Settles. 2009. Active learning literature survey. (2009).
[40]
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. 2015. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems 28 (2015).
[41]
Dhruv Verma, Sejal Bhalla, Dhruv Sahnan, Jainendra Shukla, and Aman Parnami. 2021. ExpressEar: Sensing Fine-Grained Facial Expressions with Earables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 3 (2021), 1--28.
[42]
Sergey Volokhin and Eugene Agichtein. 2018. Understanding music listening intents during daily activities with implications for contextual music recommendation. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval. 313--316.
[43]
Chen Wang and Pablo Cesar. 2017. The Play Is a Hit: But How Can You Tell?. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition. 336--347.
[44]
Xinxi Wang, David Rosenblum, and Ye Wang. 2012. Context-aware mobile music recommendation for daily activities. In Proceedings of the 20th ACM international conference on Multimedia. 99--108.
[45]
Yi-Hsuan Yang and Yuan-Ching Teng. 2015. Quantitative study of music listening behavior in a smartphone context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 3 (2015), 1--30.

Index Terms

  1. GrooveMeter: Enabling Music Engagement-aware Apps by Detecting Reactions to Daily Music Listening via Earable Sensing

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '23: Proceedings of the 31st ACM International Conference on Multimedia
    October 2023
    9913 pages
    ISBN:9798400701085
    DOI:10.1145/3581783
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 27 October 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. earable sensing
    2. music-listening engagement
    3. reaction detection

    Qualifiers

    • Research-article

    Conference

    MM '23
    Sponsor:
    MM '23: The 31st ACM International Conference on Multimedia
    October 29 - November 3, 2023
    Ottawa ON, Canada

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 200
      Total Downloads
    • Downloads (Last 12 months)98
    • Downloads (Last 6 weeks)9
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media