ARGo: An Architecture for Sign Language Recognition and Interpretation

Abstract

This paper presents a recognition and interpretation architecture dedicated to Sign Language.

A sign is composed of several co-occurring parameters that allows several heterogeneous bits of information to be emitted simultaneously depending on the variation of one of these parameters. Sign languages vary from one country to another and each has a specific vocabulary. These signs are called conventional signs and can be listed in dictionaries. A second kind of signs is extremely frequent in sign language communications: the non-conventional signs. They are created during discourse, depending on need and context, and cannot be listed in dictionaries. Moreover, some conventional signs may have one or more variable parameters, depending on context. These signs are named Variable signs.

Sign language functioning is based on simultaneousness of information and spatial rules governing sign relationships, and the vocabulary is not completely known a priori. For these reasons, classical sequential treatments are not sufficient for sign recognition. Our architecture tries to take into account such a functioning. It is composed of recognition and interpretation modules. The first module is based on Hidden Markov Models and allows us to classify conventional, non-conventional and variable signs. In the interpretation module, a virtual scene allows to store context and to complete the meaning of variable and non-conventional signs.

Because of its ability to deal with non-conventional signs, this system can be used both for sign language and co-verbal gestures.