Renjie Huang
ABSTRACT
At present, most of the melody generation models consider the introduction of chord, rhythm and other constraints in the melody generation process to ensure the quality of the melody generation. While all of them ignore the importance of emotion in melody generation. Music is an emotional art. As the primary part of a piece of music, melody usually has a clear emotional expression. Therefore, it is necessary to introduce emotion information and constraints to generate a melody with clear emotional expression, which means the model should have the ability to learn the relevant characteristics of emotions according to the given information and constraints. To this end, we propose a melody generation model ECMG with emotion constraints. The model takes Generative Adversarial Network (GAN) as the main body, and adds emotion encoder and emotion classifier to introduce emotion information and emotional constraints. We conducted quality evaluation and emotion evaluation of the melody generated by ECMG. In the evaluation of quality, the quality score difference between the melody generated by ECMG and the real melody in the training set is within 0.2, and the quality score of the melody generated by PopMNet is also relatively close. In the evaluation of emotion, the accuracy of emotion classification for both four-category and two-category is much higher than that of completely random probability. These evaluation results show that ECMG can generate melody with specific emotions while ensuring a high quality of generation.
- Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[J]. Advances in neural information processing systems, 2014, 27.Google Scholar
- Yu Y, Harscoët F, Canales S, et al. Lyrics-Conditioned Neural Melody Generation[C]//International Conference on Multimedia Modeling. Springer, Cham, 2020: 709--714.Google Scholar
- Zhu H, Liu Q, Yuan N J, et al. Xiaoice band: A melody and arrangement generation framework for pop music[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018: 2837--2846.Google Scholar
- Chen Y, Lerch A. Melody-Conditioned Lyrics Generation with SeqGANs[J]. arXiv preprint arXiv:2010.14709, 2020.Google Scholar
- Yuan R, Zhang G, Yang A, et al. Diverse Melody Generation from Chinese Lyrics via Mutual Information Maximization[J]. arXiv preprint arXiv:2012.03805, 2020.Google Scholar
- Elliot Waite, Douglas Eck, Adam Roberts, and Dan Abolafia. Project Magenta: Generating long-term structure in songs and stories, 2016. https://magenta.tensorflow.org/blog/2016/07/15/lookback-rnn-attention-rnn/.Google Scholar
- Yang L C, Chou S Y, Yang Y H. MidiNet: A convolutional generative adversarial network for symbolic-domain music generation[J]. arXiv preprint arXiv:1703.10847, 2017.Google Scholar
- Scirea M, Togelius J, Eklund P, et al. Affective evolutionary music composition with MetaCompose[J]. Genetic Programming and Evolvable Machines, 2017, 18(4): 433--465.Google ScholarDigital Library
- Roberts A, Engel J, Raffel C, et al. A hierarchical latent vector model for learning long-term structure in music[J]. arXiv preprint arXiv:1803.05428, 2018.Google Scholar
- Yu Y, Srivastava A, Canales S. Conditional lstm-gan for melody generation from lyrics[J]. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2021, 17(1): 1--20.Google Scholar
- Wu J, Liu X, Hu X, et al. PopMNet: Generating structured pop music melodies using neural networks[J]. Artificial Intelligence, 2020, 286: 103303.Google ScholarCross Ref
- Laurier C, Herrera P, Mandel M, et al. Audio music mood classification using support vector machine[J]. MIREX task on Audio Mood Classification, 2007: 2--4.Google Scholar
- Panda R E S, Malheiro R, Rocha B, et al. Multi-modal music emotion recognition: A new dataset, methodology and comparative analysis[C]//10th International Symposium on Computer Music Multidisciplinary Research (CMMR 2013). 2013: 570--582.Google Scholar
- Russell J A. Core affect and the psychological construction of emotion[J]. Psychological review, 2003, 110(1): 145.Google Scholar
- Yang Y H, Lin Y C, Su Y F, et al. A regression approach to music emotion recognition[J]. IEEE Transactions on audio, speech, and language processing, 2008, 16(2): 448--457.Google ScholarDigital Library
- Lin Y, Chen X, Yang D. Exploration of Music Emotion Recognition Based on MIDI[C]//ISMIR. 2013: 221--226.Google Scholar
- Delbouys R, Hennequin R, Piccoli F, et al. Music mood detection based on audio and lyrics with deep neural net[J]. arXiv preprint arXiv:1809.07276, 2018.Google Scholar
- Lin J C, Wei W L, Wang H M. Automatic music video generation based on emotion-oriented pseudo song prediction and matching[C]//Proceedings of the 24th ACM international conference on Multimedia. 2016: 372--376.Google Scholar
- Madhok R, Goel S, Garg S. SentiMozart: Music Generation based on Emotions[C]//ICAART (2). 2018: 501--506.Google Scholar
- Liu R, Yuan S, Chen M, et al. MaLiang: An Emotion-driven Chinese Calligraphy Artwork Composition System[C]//Proceedings of the 28th ACM International Conference on Multimedia. 2020: 4394--4396.Google Scholar
- Hou Y, Yao H, Sun X, et al. Soul Dancer: Emotion-Based Human Action Generation[J]. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2020, 15(3s): 1--19.Google ScholarDigital Library
- Raffel C. Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching[M]. Columbia University, 2016.Google Scholar
- Aljanaki A, Yang Y H, Soleymani M. Developing a benchmark for emotional analysis of music[J]. PloS one, 2017, 12(3): e0173392.Google Scholar
- Panda R, Malheiro R, Paiva R P. Novel audio features for music emotion recognition[J]. IEEE Transactions on Affective Computing, 2018, 11(4): 614--626.Google ScholarDigital Library
- Panda R, Malheiro R, Paiva R P. Musical texture and expressivity features for music emotion recognition[C]//19th International Society for Music Information Retrieval Conference (ISMIR 2018. 2018: 383--391.Google Scholar
Index Terms
- Melody Generation with Emotion Constraint
Recommendations
Functional generation of harmony and melody
FARM '14: Proceedings of the 2nd ACM SIGPLAN international workshop on Functional art, music, modeling & designWe present FComp, a system for automatic generation of harmony and accompanying melody. Building on previous work on functional modelling of musical harmony, FComp first creates a foundational harmony by generating random (but user-guided) values of a ...
Emotion control system for MIDI excerpts: MOR2ART
Fun and Games '10: Proceedings of the 3rd International Conference on Fun and GamesEmotional expression when performing music (singing or playing musical instruments) requires skill, but such a skill is generally difficult to learn. Computer systems that can make it easy for non-musicians to express any emotion have been proposed[1]. ...
Do all facial emojis communicate emotion? The impact of facial emojis on perceived sender emotion and text processing
AbstractFacial emojis can express a variety of positive and negative emotions, and are commonly used in digital, written communication. However, little is known about how emojis impact text processing and how different emoji-text combinations ...
Highlights- We investigated how facial emojis impact text comprehension using behavioral ratings and EEG.
Comments