Hierarchical attentive deep neural networks for semantic music annotation through multiple music representations

Abstract

Automatically assigning a group of appropriate semantic tags to one music piece provides an effective way for people to efficiently utilize the massive and ever increasing online and off-line music data. In this paper, we propose a novel end-to-end deep neural network model for automatic music annotation, which effectively integrates available multiple complementary music representations and jointly accomplishes music representation learning, structure modeling, and tag prediction. The model first hierarchically leverages attentive convolutional networks and recurrent networks to learn informative descriptions from Mel-spectrogram and raw waveform of the music and depict time-varying structures embedded in the description sequence. A dual-state LSTM network is then employed to capture the correlations between two representation channels as supplementary music descriptions. Finally, the model aggregates music description sequence into a holistic embedding with a self-attentive multi-weighting mechanism, which adaptively captures multi-aspect summarized information of the music for tag prediction. Experiments on the public MagnaTagATune benchmark music dataset show that the proposed model outperforms state-of-the-art methods for automatic music annotation.