Recognition of Drum Music Using Sound and Video

Abstract

Drums are one of the important instruments in music, especially in rock and pop music. At present, many videos of playing drums have been posted on video sites. These performance videos are viewed not only for the enjoyment of watching, but also for the benefit of improving performance skills, especially for intermediate and advanced drummers. However, for beginners of drumming, it is often difficult to refer to the performance video because the performance movement is too fast or it is invisible behind instruments or other objects. Therefore, there is a great need for musical scores corresponding to performances in videos. The musical scoring was usually done by people with experience and ability, but in recent years, research has been conducted on the automation of the musical scoring, especially from sound sources, for various musical instruments such as pianos. However, there are few studies on musical scoring for a drum set. This is because it is difficult to identify drum sounds by frequency analysis. In this paper, we propose a method to increase the accuracy of the musical scoring by using both music and video of a drum performance. In addition, we conduct performance evaluation experiments on actual drum performances using the system implementing the proposed method, and we show the effectiveness of the proposed method.