Detection of Pointing Position by Omnidirectional Camera

Abstract

In recent years, voice recognition devices such as smart speakers have been in the limelight, but intuitive communication such as instructing the target with ambiguous expressions is difficult with voice alone. In order to realize such communication, it is effective to combine gestures such as pointing with voice. Therefore, we propose a method to detect the pointing position from the image of the omnidirectional camera using a convolutional neural network. Although many methods have been proposed to detect the pointing position using a normal camera, the standing position of the person who gives instructions is limited since the observation area is small. We solve this problem by using an omnidirectional camera. First, the proposed method converts a hemisphere image taken from an omnidirectional camera to a panoramic image. Next, the bounding box surrounding the person with pointing gesture is detected in the panoramic image by the object detection network. Finally, the pointing position estimation network estimates the pointing position in the panoramic image from the image in the bounding box and its location. Since it is difficult to prepare a large number of pointing gesture images, CG images created by Unity are used for pre-training. Experiments using real images of pointing gesture shows that the proposed method is effective for pointing position detection.