Generation of Brass Band Animation Synchronized with the Motion of Conductor’s Hand

Abstract

In recent years, VR devices such as PlayStation VR, HTC Vive, and Oculus Rift have become widespread, making it possible to easily experience Virtual Reality. There are contents that can get a high immersive feeling by synchronizing the motion of the user with the CG and music in the virtual space. Research on automatic generation of musical instrument performance animations using MIDI data, which is a type of music data, has been conducted. However, there is a problem that the motion and the sound during the performance are shifted, giving the viewer a sense of incongruity.

In this research, we propose a generation system of brass band animation synchronized with the motion of conductor’s hand using VR devices. The user uses the VR controller as a baton. The user can control music and animation by performing the action of the conductor. In our system, the motion of the conductor’s hand is acquired, and the motion speed and amount of motion are calculated. After that, animation and music are controlled using the acquired data. Specifically, the point at which the direction of the acquired velocity vector changes is set as the beat start point. Then, the music speed and animation are controlled using the estimated tempo. In addition, the music volume is changed using the calculated amount of motion. Furthermore, by generating CG animation in the virtual space, it gives the user a sense of immersion. As an experiment, 12 people used this system and verified the usefulness of the method.