skip to main content
10.1145/3631700.3665196acmconferencesArticle/Chapter ViewAbstractPublication PagesumapConference Proceedingsconference-collections
research-article
Open access

EmoSynth Real Time Emotion-Driven Sound Texture Synthesis via Brain-Computer Interface

Published: 28 June 2024 Publication History

Abstract

In electroacoustic music composition, particularly in sound synthesis techniques, Deep Learning (DL) provides very effective solutions. However, these architectures generally have a high level of automation and use textual language for human interaction. To improve the relationship between composers and artificial intelligence systems, brain-computer interfaces (BCIs) are an effective and direct systems, which have led to considerable improvements in this area. The proposed system employs emotion recognition through electroencephalogram (EEG) signals to control four Variational Autoencoders (VAE) that generate new sound textures. A dataset was acquired using the MUSE2 headset to train four Machine Learning (ML) models capable of classifying human emotions based on Russell’s circumplex model. VAEs were trained to produce different sound variations from an audio dataset that allows composers to integrate their sounds. In addition, a graphical user interface (GUI) was developed to facilitate the real-time generation of sound textures, with the support of an external MIDI controller. This GUI continuously provides visual information about the detected emotions and the activity of the left and right brain hemispheres.

References

[1]
Carmelo Ardito, Ilaria Bortone, Tommaso Colafiglio, Tommaso Di Noia, Eugenio Di Sciascio, Domenico Lofù, Fedelucio Narducci, Rodolfo Sardone, and Paolo Sorino. 2022. Brain computer interface: Deep learning approach to predict human emotion recognition. In 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2689–2694.
[2]
N Barascud. [n. d.]. meegkit: EEG and Meg denoising in Python.
[3]
Pietro Boccadoro, Vitanio Daniele, Pietro Di Gennaro, Domenico Lofù, and Pietro Tedeschi. 2022. Water quality prediction on a sigfox-compliant iot device: The road ahead of waters. Ad Hoc Networks 126 (2022), 102749.
[4]
Tommaso Colafiglio, Domenico Lofù, Paolo Sorino, Fabrizio Festa, Tommaso Di Noia, and Eugenio Di Sciascio. 2023. Exploring the Mental State Intersection by Brain-Computer Interfaces, Cellular Automata and Biofeedback. In IEEE EUROCON 2023-20th International Conference on Smart Technologies. IEEE, 461–466.
[5]
Tommaso Colafiglio, Paolo Sorino, Domenico Lofu, Angela Lombardi, Fedelucio Narducci, and Tommaso Di Noia. 2023. Combining Mental States Recognition and Machine Learning for Neurorehabilitation. In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 3848–3853.
[6]
Sander Dieleman, Aaron Van Den Oord, and Karen Simonyan. 2018. The challenge of realistic music generation: modelling raw audio at scale. Advances in neural information processing systems 31 (2018).
[7]
Antonio Gulli and Sujit Pal. 2017. Deep learning with Keras. Packt Publishing Ltd.
[8]
Jiachen Hu Hu, Xuebin Qin Qin, and Peijiao Yang Yang. 2022. Emotion recognition based on visual evoked EEG. In International Conference on Intelligent Manufacturing and Industrial Automation (CIMIA 2022), Vol. 12289. SPIE, 87–90.
[9]
Yu-Siang Huang and Yi-Hsuan Yang. 2020. Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. In Proceedings of the 28th ACM international conference on multimedia. 1180–1188.
[10]
Valton Kamberaj, Arbana Kadriu, and Nuhi Besimi. 2023. Instruments Music Composition in Different Genres and Techniques Using AI: A Review. In International Scientific Conference on Business and Economics. Springer, 741–754.
[11]
Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. 2011. Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing 3, 1 (2011), 18–31.
[12]
Ningning Liu, Kai Wang, Xin Jin, Boyang Gao, Emmanuel Dellandrea, and Liming Chen. 2017. Visual affective classification by combining visual and text features. PloS one 12, 8 (2017), e0183018.
[13]
Domenico Lofù, Pietro Di Gennaro, Pietro Tedeschi, Tommaso Di Noia, and Eugenio Di Sciascio. 2023. URANUS: Radio Frequency Tracking, Classification and Identification of Unmanned Aircraft Vehicles. IEEE Open Journal of Vehicular Technology (2023).
[14]
Domenico Lofù, Andrea Pazienza, Carmelo Ardito, Tommaso Di Noia, Eugenio Di Sciascio, and Felice Vitulano. 2022. A situation awareness computational intelligent model for metabolic syndrome management. In 2022 IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA). IEEE, 118–124.
[15]
Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and music signal analysis in python. In SciPy. 18–24.
[16]
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. 2016. SampleRNN: An unconditional end-to-end neural audio generation model. arXiv preprint arXiv:1612.07837 (2016).
[17]
Guanxiong Pei, Qian Shang, Shizhen Hua, Taihao Li, and Jia Jin. 2024. EEG-based affective computing in virtual reality with a balancing of the computational efficiency and recognition accuracy. Computers in Human Behavior 152 (2024), 108085.
[18]
Denis Rothman. 2021. Transformers for Natural Language Processing: Build innovative deep neural network architectures for NLP with Python, PyTorch, TensorFlow, BERT, RoBERTa, and more. Packt Publishing Ltd.
[19]
James A Russell. 1980. A circumplex model of affect.Journal of personality and social psychology 39, 6 (1980), 1161.
[20]
Paolo Sorino, Vincenzo Paparella, Domenico Lofu, Tommaso Colafiglio, Eugenio Di Sciascio, Fedelucio Narducci, Rodolfo Sardone, and Tommaso Di Noia. 2023. A Pareto-Optimality-based approach for selecting the best Machine Learning models in mild cognitive impairment prediction. In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 3822–3827.
[21]
Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu, 2016. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 12 (2016).
[22]
Wei Xu, Ruoxuan Zhou, and Qiuming Liu. 2022. Electroencephalogram Emotion Recognition Based on Three-Dimensional Feature Matrix and Multivariate Neural Network. In 2022 IEEE 25th International Conference on Computational Science and Engineering (CSE). IEEE, 32–37.

Index Terms

  1. EmoSynth Real Time Emotion-Driven Sound Texture Synthesis via Brain-Computer Interface

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UMAP Adjunct '24: Adjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization
    June 2024
    662 pages
    ISBN:9798400704666
    DOI:10.1145/3631700
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 28 June 2024

    Check for updates

    Author Tags

    1. Artificial Intelligence
    2. Brain-Machine Interface
    3. Explainable AI.
    4. Neural Instrument

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    UMAP '24
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 162 of 633 submissions, 26%

    Upcoming Conference

    UMAP '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 242
      Total Downloads
    • Downloads (Last 12 months)242
    • Downloads (Last 6 weeks)37
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media