Recognition of Listener’s Nodding by LSTM Based on Movement of Facial Keypoints and Speech Intonation

Abstract

Communication between humans and robots is crucial to achieve successful cooperation in real-life scenarios. The robot must understand not only linguistic expressions, but also non-linguistic expressions such as nodding and gestures. In this research, we examine whether a listener nods in response to a speaker’s utterance. Our proposed method judges nodding based on the movement of the listener’s facial keypoints and the speaker’s speech intonation. The proposed method achieves approximately 84.4% recognition accuracy when we input the movement and intonation simultaneously. This improves nodding recognition accuracy by 8.8% over movement only approach. This result indicates that the movement of the listener’s facial keypoints and the speaker’s intonation are important information in nodding recognition.