Expert system for gesture recognition in terminal's user interface
Introduction
The target of the gesture recognition as a part of user interface research of portable terminals is to replace different kinds of keyboard functions with gestures, i.e. with movements. In the user interface research of portable mobile terminals the replacement of traditional keyboard functions with controlled movements is especially important in very small and simple devices without keyboard and screen. Furthermore, it is very useful in ‘normal’ size devices with keyboard and screen, like mobile phones and PDA (Personal Digital Assistant) devices, as an optional choice for the traditional user interface. For example, the incoming calls can be initiated via lifting the phone to the ear and in the same way hang up via transferring it back to the table or pocket without pressing any keys or giving voice commands. In the same way the different menu options, as an example, on the PDA device can be chosen via different kind of menu specific movements of the device. However, the gesture or movement recognition has several problems like the unreliability of recognised gestures and quite heavy computational load needed for the recognition as well as high data sample rate requirement. Usually the gesture recognition is performed via filtering raw time series data and using hidden Markov chain (HMM) modelling. Even if this method is quite reliable at the high data sample rate (frequency around 80 Hz), it is computationally heavy and quite a slow. Therefore, it is not optimal for the embedded real time systems with very limited resources like low data sample rate, low computational resources, limited operating and standby times of batteries and high delay/response time due to complex data processing.
In this application the gestures are composed of time series data of three acceleration sensors integrated to the portable terminal. Acceleration sensors are positioned on 90° angle with each other in order to get three-dimensional (3D) voltage signal, i.e. acceleration vector, For the comparative HMM method the vector is filtered, normalised and quantised. For the other comparative method, fuzzy c-means classification, different features are extracted from the vector and used as an input vector whereas for the developed fuzzy rule aided classification and Self Organising Map, SOM, classification the vector is autocorrelated and Fourier transformed to get 3D Doppler spectrum from the movement. The relative maximum values (compared to the Doppler spread) of the Doppler spectrum are then used as input values to a fuzzy rule aided reasoning module. The fuzzy reasoning recognises different gestures according to the relative maximum values of Doppler spectrum. Different data preprocessing methods for different classification methods were selected according to the best results achieved for the used method. Therefore, in the final comparison we have included only the best combinations of the data preprocessing and classification methods. In Fig. 1 has presented a simplified logical architecture of the research and in Fig. 2 has presented a simplified architecture of the embedded expert system used for the gesture recognition.
The organisation of the rest of this paper is following. Section 2 briefly summarizes the basic principles of the fuzzy set theory and fuzzy logic used in the inference process in this application. The section also illustrates fuzzy methods and techniques used in the model including numerical presentation of rule base, numerical equation form reasoning and fuzzy c-means classification with the description of the logical structure and functions of the developed model, too. Section 3 describes and illustrates shortly the used hidden Markov modelling (HMM) as a part of gesture recognition. Section 4 presents basic principles of the Self Organising Map, SOM, and describes used learning parameters and equations. Section 5 presents the figures of acceleration vectors and the Doppler spectrums of the example gestures and the implementation environment of the classification model. Results and discussions of the selected approaches are presented in Section 6. Finally conclusions are drawn in Section 7.
Section snippets
Fuzzy set theory and fuzzy logic
Fuzzy set theory was originally presented by L. Zadeh in his seminal paper ‘Fuzzy Sets’ Zadeh (1965). Fuzzy logic was developed later on from it to reason with uncertain and vague information and to represent knowledge in operationally powerful form.
The name fuzzy sets are used to distinguish them from the crisp sets of the conventional set theory. The characteristic function of a crisp set C, μC(u), assigns a discrete value1 to each element u in the universal set U. The
Gesture recognition with HMM
Markov model is a statistical model used for characterizing the properties of a given signal. Output of the Markov process is a set of states at each instant of time, where each state corresponds to a physical (observable) event. However, this model is not sufficient to be applicable to many problems of real world. Concept of Markov Model can be extended to include the case, where the observation is a probabilistic function of the state. The resulting model is a doubly stochastic process with
Self-organising map (SOM)
SOM, is a two layered neural network. It can organise a topological map from a random initial point showing the natural relationships among the input patterns given to the network. In other words, it finds the structure of relationships among input patterns, which are classified by the units they activate in the competitive layer. The SOM network combines an input layer with a competitive layer. It is trained by unsupervised learning and it provides a graphical oraganisation of pattern
System model
A simplified architecture of research arrangement has been described in Fig. 1. Three acceleration sensors were embedded into a mobile terminal according to the direction of rectangular co-ordinate axes (x, y, z in Fig. 1. The acceleration data was sampled at the frequency of 20 Hz for each sensors. The acceleration data vector was autocorrelated and Fourier transformed (see Fig. 2, where preprocessing refers to Doppler spectrum and Doppler spread definitions) in order to get (maximum
Results
The main motivation of this research was to find out reliable and computationally light method for embedded expert systems to replace computationally heavy methods, like HMM, in gesture recognition on the user interface research. Therefore, we compared our method against fuzzy c-means FCM, SOM and the HMM methods. Results of the recognition can be seen in Table 2.
The HMM results are not at the same reliability level than in the other methods. This mainly due to fact that acceleration vector is
Conclusions
In this paper we described embedded expert system as a part of intelligent user interface of a mobile terminal for gesture recognition procedure. We compared the developed fuzzy rule aided classification method of the expert system to the fuzzy c-means classification, HMM classification and SOM classification methods. The developed embedded expert system increases significantly reliability of gesture recognition/classification. The other advantages of fuzzy logic based gesture recognition
Acknowledgements
Technical Research Centre of Finland is acknowledged for the finance of research.
References (10)
- et al.
Fuzzy logic based forecasting model
Engineering Applications of Artificial Intelligence
(2001) Fuzzy sets
Information and Control
(1965)Pattern recognition with fuzzy objective function
(1981)- et al.
An introduction to fuzzy control
(1996) - et al.
Velocity profile based recognituon of dynamic gestures with discrete hidden Markov models
(1997)
Cited by (12)
Medical gesture recognition using dynamic arc length warping
2019, Biomedical Signal Processing and ControlCitation Excerpt :In this particular analysis, the complete medical procedure is decomposed and segmented into smaller units. Under such approach, measures of motor performance are recorded including kinematic and kinetic measurements, such as position, velocity, acceleration, and force/torque values [25,9]. Results, obtained in this field, have shown, in both open and MIS (Minimally Invasive Surgery) simulators, a correlation between movements made and objective skill measurements in simulation (Spearman coefficient 0.53) [6,7].
Handy: A real-time three color glove-based gesture recognizer with learning vector quantization
2012, Expert Systems with ApplicationsCitation Excerpt :The paper is organized as follows: Section 2 describes the approach used; Section 3 gives an account of the segmentation module; the feature extraction process is discussed in Section 4; a review of learning vector quantization is provided in Section 5; Section 6 reports some experimental results; in Section 7 some conclusions are drawn. Several approaches were proposed for gesture recognition (Chaudhary, Raheja, Das, & Raheja, 2011; Frantti & Kallio, 2004; Huang, Hu, & Chang, 2011; Mitra & Acharya, 2007; Tsai & Lee, 2011). Our approach was inspired by virtual reality applications (Burdea & Coiffet, 2003) where the movements of the hands of people are tracked asking them to wear data gloves (Dipietro, Sabatini, & Dario, 2008).
Analysis of pattern recognition techniques for in-air signature biometrics
2011, Pattern RecognitionExpert system methodologies and applications-a decade review from 1995 to 2004
2005, Expert Systems with Applications