Expert system for gesture recognition in terminal's user interface

https://doi.org/10.1016/S0957-4174(03)00134-9Get rights and content

Abstract

This paper presents and describes a soft computing based expert system for gesture recognition procedure, as a part of intelligent user interface of a mobile terminal. In the presented solution, a terminal includes three acceleration sensors positioned like xyz co-ordinate system in order to get three-dimensional (3D) acceleration vector, xyz. The 3D acceleration vector is, after Doppler spectrum definition, used as an input vector to a fuzzy reasoning unit of embedded expert system, which classifies gestures (time series of acceleration vectors). In the reasoning unit fuzzy rule aided method is used to classification. The method is compared to the fuzzy c-means classification with feature extraction, to the hidden Markov model (HMM) classification and SOM classification. Fuzzy methods classified successfully the test sets. The advantages of the fuzzy methods are computational effectiveness, simple implementation, lower data sample rate requirement and reliability. Moreover, fuzzy methods do not require training like SOM and HMM. Therefore, the methods can be applied to the real time systems where different gestures can be used, for example, instead of the keyboard functions. The computational effectiveness and low sample rate requirement also increases the operational time of device compared to computationally heavy HMM method. Furthermore, the easy implementation and reliability are important factors for the success of the new technology's spreading on the mass market of terminals.

Introduction

The target of the gesture recognition as a part of user interface research of portable terminals is to replace different kinds of keyboard functions with gestures, i.e. with movements. In the user interface research of portable mobile terminals the replacement of traditional keyboard functions with controlled movements is especially important in very small and simple devices without keyboard and screen. Furthermore, it is very useful in ‘normal’ size devices with keyboard and screen, like mobile phones and PDA (Personal Digital Assistant) devices, as an optional choice for the traditional user interface. For example, the incoming calls can be initiated via lifting the phone to the ear and in the same way hang up via transferring it back to the table or pocket without pressing any keys or giving voice commands. In the same way the different menu options, as an example, on the PDA device can be chosen via different kind of menu specific movements of the device. However, the gesture or movement recognition has several problems like the unreliability of recognised gestures and quite heavy computational load needed for the recognition as well as high data sample rate requirement. Usually the gesture recognition is performed via filtering raw time series data and using hidden Markov chain (HMM) modelling. Even if this method is quite reliable at the high data sample rate (frequency around 80 Hz), it is computationally heavy and quite a slow. Therefore, it is not optimal for the embedded real time systems with very limited resources like low data sample rate, low computational resources, limited operating and standby times of batteries and high delay/response time due to complex data processing.

In this application the gestures are composed of time series data of three acceleration sensors integrated to the portable terminal. Acceleration sensors are positioned on 90° angle with each other in order to get three-dimensional (3D) voltage signal, i.e. acceleration vector, xyz. For the comparative HMM method the xyz vector is filtered, normalised and quantised. For the other comparative method, fuzzy c-means classification, different features are extracted from the vector and used as an input vector whereas for the developed fuzzy rule aided classification and Self Organising Map, SOM, classification the vector is autocorrelated and Fourier transformed to get 3D Doppler spectrum from the movement. The relative maximum values (compared to the Doppler spread) of the Doppler spectrum are then used as input values to a fuzzy rule aided reasoning module. The fuzzy reasoning recognises different gestures according to the relative maximum values of Doppler spectrum. Different data preprocessing methods for different classification methods were selected according to the best results achieved for the used method. Therefore, in the final comparison we have included only the best combinations of the data preprocessing and classification methods. In Fig. 1 has presented a simplified logical architecture of the research and in Fig. 2 has presented a simplified architecture of the embedded expert system used for the gesture recognition.

The organisation of the rest of this paper is following. Section 2 briefly summarizes the basic principles of the fuzzy set theory and fuzzy logic used in the inference process in this application. The section also illustrates fuzzy methods and techniques used in the model including numerical presentation of rule base, numerical equation form reasoning and fuzzy c-means classification with the description of the logical structure and functions of the developed model, too. Section 3 describes and illustrates shortly the used hidden Markov modelling (HMM) as a part of gesture recognition. Section 4 presents basic principles of the Self Organising Map, SOM, and describes used learning parameters and equations. Section 5 presents the figures of acceleration vectors and the Doppler spectrums of the example gestures and the implementation environment of the classification model. Results and discussions of the selected approaches are presented in Section 6. Finally conclusions are drawn in Section 7.

Section snippets

Fuzzy set theory and fuzzy logic

Fuzzy set theory was originally presented by L. Zadeh in his seminal paper ‘Fuzzy Sets’ Zadeh (1965). Fuzzy logic was developed later on from it to reason with uncertain and vague information and to represent knowledge in operationally powerful form.

The name fuzzy sets are used to distinguish them from the crisp sets of the conventional set theory. The characteristic function of a crisp set C, μC(u), assigns a discrete value1 to each element u in the universal set U. The

Gesture recognition with HMM

Markov model is a statistical model used for characterizing the properties of a given signal. Output of the Markov process is a set of states at each instant of time, where each state corresponds to a physical (observable) event. However, this model is not sufficient to be applicable to many problems of real world. Concept of Markov Model can be extended to include the case, where the observation is a probabilistic function of the state. The resulting model is a doubly stochastic process with

Self-organising map (SOM)

SOM, is a two layered neural network. It can organise a topological map from a random initial point showing the natural relationships among the input patterns given to the network. In other words, it finds the structure of relationships among input patterns, which are classified by the units they activate in the competitive layer. The SOM network combines an input layer with a competitive layer. It is trained by unsupervised learning and it provides a graphical oraganisation of pattern

System model

A simplified architecture of research arrangement has been described in Fig. 1. Three acceleration sensors were embedded into a mobile terminal according to the direction of rectangular co-ordinate axes (x, y, z in Fig. 1. The acceleration data was sampled at the frequency of 20 Hz for each sensors. The acceleration data vector xyz was autocorrelated and Fourier transformed (see Fig. 2, where preprocessing refers to Doppler spectrum and Doppler spread definitions) in order to get (maximum

Results

The main motivation of this research was to find out reliable and computationally light method for embedded expert systems to replace computationally heavy methods, like HMM, in gesture recognition on the user interface research. Therefore, we compared our method against fuzzy c-means FCM, SOM and the HMM methods. Results of the recognition can be seen in Table 2.

The HMM results are not at the same reliability level than in the other methods. This mainly due to fact that acceleration vector is

Conclusions

In this paper we described embedded expert system as a part of intelligent user interface of a mobile terminal for gesture recognition procedure. We compared the developed fuzzy rule aided classification method of the expert system to the fuzzy c-means classification, HMM classification and SOM classification methods. The developed embedded expert system increases significantly reliability of gesture recognition/classification. The other advantages of fuzzy logic based gesture recognition

Acknowledgements

Technical Research Centre of Finland is acknowledged for the finance of research.

References (10)

  • T. Frantti et al.

    Fuzzy logic based forecasting model

    Engineering Applications of Artificial Intelligence

    (2001)
  • L. Zadeh

    Fuzzy sets

    Information and Control

    (1965)
  • J. Bezdek

    Pattern recognition with fuzzy objective function

    (1981)
  • D. Driankov et al.

    An introduction to fuzzy control

    (1996)
  • F. Hoffman et al.

    Velocity profile based recognituon of dynamic gestures with discrete hidden Markov models

    (1997)
There are more references available in the full text version of this article.

Cited by (12)

  • Medical gesture recognition using dynamic arc length warping

    2019, Biomedical Signal Processing and Control
    Citation Excerpt :

    In this particular analysis, the complete medical procedure is decomposed and segmented into smaller units. Under such approach, measures of motor performance are recorded including kinematic and kinetic measurements, such as position, velocity, acceleration, and force/torque values [25,9]. Results, obtained in this field, have shown, in both open and MIS (Minimally Invasive Surgery) simulators, a correlation between movements made and objective skill measurements in simulation (Spearman coefficient 0.53) [6,7].

  • Handy: A real-time three color glove-based gesture recognizer with learning vector quantization

    2012, Expert Systems with Applications
    Citation Excerpt :

    The paper is organized as follows: Section 2 describes the approach used; Section 3 gives an account of the segmentation module; the feature extraction process is discussed in Section 4; a review of learning vector quantization is provided in Section 5; Section 6 reports some experimental results; in Section 7 some conclusions are drawn. Several approaches were proposed for gesture recognition (Chaudhary, Raheja, Das, & Raheja, 2011; Frantti & Kallio, 2004; Huang, Hu, & Chang, 2011; Mitra & Acharya, 2007; Tsai & Lee, 2011). Our approach was inspired by virtual reality applications (Burdea & Coiffet, 2003) where the movements of the hands of people are tracked asking them to wear data gloves (Dipietro, Sabatini, & Dario, 2008).

View all citing articles on Scopus
View full text