Automatic hand motion analysis for the sign language space management

Abstract

In this paper, we present a new approach for the sign language (SL) space management based on hand motion analysis from RGB-D data. The aim of this work is to extract the equation of the hand motion curve in order to provide a parametric description of the sign. This description enables the SL space management by changing dynamically the entities locations, directions and motion velocity according to the SL sentence context. Our main contribution involves three modules: the automatic motion curve approximation, the automatic extraction of the motion signature (identity) and the automatic sign language space segmentation. The first module aims to apply regression techniques for approximating the motion curves of the two hands. In the second module, we exploited the Gaussian mixture models (GMM) to classify the different motion samples of the same sign based on the homogeneity constraint. The goal of this process is to eliminate the noise configurations and to extract the appropriate curve identity by choosing the cluster having the biggest votes. The average value of all the motion parameters in this cluster leads us to determine the movement identity of the sign. In the third module, we used GMM for the automatic sign space segmentation based on the highest density areas. Our motion analysis approach is experimented on 500 annotated American Sign Language signs and assessed by $R^{2}$ and MSE metrics for evaluating the approximation quality. We obtained good experimental results with 93% overall satisfaction rate with 468 signs having $R^{2}\ge 0.85$. We improved the curve approximation of the rest of signs (32 signs) to reach $R^{2}\ge 0.85$ by using our motion segmentation approach.