Finger identification and hand posture recognition for human–robot interaction

doi:10.1016/j.imavis.2006.08.003

Image and Vision Computing

Volume 25, Issue 8, 1 August 2007, Pages 1291-1300

https://doi.org/10.1016/j.imavis.2006.08.003 Get rights and content

Abstract

Natural and friendly interface is critical for the development of service robots. Gesture-based interface offers a way to enable untrained users to interact with robots more easily and efficiently. In this paper, we present a posture recognition system implemented on a real humanoid service robot. The system applies RCE neural network based color segmentation algorithm to separate hand images from complex backgrounds. The topological features of the hand are then extracted from the silhouette of the segmented hand region. Based on the analysis of these simple but distinctive features, hand postures are identified accurately. Experimental results on gesture-based robot programming demonstrated the effectiveness and robustness of the system.

Introduction

With the massive influx of computers in society and the increasing importance of service sectors in many of industrialized nations, the market for robots in conventional applications of manufacturing automation is reaching saturation, and the research on robotics is rapidly proliferating in the field of service industries [1], [2]. Service robots are intelligent machines that provide service for human beings and machines themselves. They operate in dynamic and unstructured environment and interact with people who are not necessarily skilled in communicating with robots [3]. Friendly and cooperative interface is thus critical for the development of service robots [4], [5]. Gesture-based interface holds the promise of making human–robot interaction more natural and efficient.

Gesture-based interaction was firstly proposed by M.W. Krueger as a new form of human–computer interaction in the middle of the seventies [6], and there has been a growing interest in it recently. As a special case of human–computer interaction, human–robot interaction is imposed by several constraints [7]: the background is complex and dynamic; the lighting condition is variable; the shape of the human hand is deformable; the implementation is required to be executed in real time and the system is expected to be user and device independent. Numerous techniques on gesture-based interaction have been proposed, but hardly any published work fulfills all the requirements stated above.

R. Kjeldsen and J. Kender [8] presented a realtime gesture system which is used in place of the mouse to move and resize windows. In this system, the hand is segmented from the background using skin color and the hand’s pose is classified using a neural network. A drawback of the system is that its hand tracking has to be specifically adapted for each user. The Perseus system developed by R.E. Kahn [9] was used to recognize the pointing gesture. In the system, a variety of features, such as intensity, edge, motion, disparity and color has been used for gesture recognition. This system is implemented only in a restricted indoor environment. In the gesture-based human–robot interaction system of J. Triesch and C. Ven Der Malsburg [7], the combination of motion, color and stereo cues was used to track and locate the human hand, and the hand posture recognition was based on elastic graph matching. This system is person independent and can work in the presence of complex backgrounds in real time. But it is prone to noise and sensitive to the change of the illumination because its skin color detection was based on a defined prototypical skin color point in the HS plane.

This paper presents a simple, fast and robust system that segment and recognize hand postures for human–robot interaction. In the system, a novel color segmentation algorithm developed on the basis of Restricted Coulomb Energy (RCE) neural network is applied to segment hand images. This method uses the skin color prototype to describe the skin color. With the abundant skin color prototypes that are derived from the training procedure of the RCE network, the system is capable of characterizing the distribution region of skin colors accurately in the color space and segment various hand images efficiently from complex backgrounds. The topological features of the hand are then extracted from the silhouette of the segmented hand region, and the recognition of hand postures is based on the analysis of these features. The system has been experimented with several postures for gesture-based robot programming and human–robot interaction on a real humanoid service robot.

The rest of the paper is organized as follows. The problem of hand image segmentation is addressed in the next section. The proposed algorithms for hand feature extraction and posture recognition are then presented in Section 3. Section 4 introduces our humanoid service robot HARO-1, illustrates the method of robot programming, and states the interaction procedure of the system. Finally, conclusions are given in Section 5.

Section snippets

Hand image segmentation

Hand image segmentation separates the hand image from the background. It is the first important step in every hand gesture recognition system, and all subsequent stages heavily rely on the quality of the segmentation. Two types of cues, color cues and motion cues, are often applied for hand image segmentation [10]. Motion cues are used in conjunction with certain assumptions [11], [12]. For example, the gesturer is stationary with respect to the background that is also stationary. Such

Feature selection

Hand segmentation is followed by feature extraction. Contour is the commonly used feature for accurate recognition of hand postures, and can be extracted easily from the silhouette of the segmented hand region. In our study, we found it is difficult to extract the smooth and continuous contour of the hand because the segmented hand region is irregular, especially when the RCE neural network is not trained sufficiently. In Fig. 4, (a) shows the segmentation of hand image, (b) shows the

Humanoid service robot

Our research on hand gesture recognition is a part of the project of Hybrid Service Robot System, in which we will integrate various technologies, such as real robot control, virtual robot simulation, human–robot interaction etc., to build a multi-modal and intelligent human–robot interface. Fig. 9(a) shows the human-alike service robot HARO-1 at our lab. It was designed and developed by ourselves, and mainly consists of an active stereo vision head on modular neck, two modular arms with active

Conclusions

We have presented a gesture recognition system implemented on a real humanoid service robot. The system applies RCE neural network to segment hand images. The RCE network is capable of characterizing the distribution region of all skin colors in color space with numerous skin color prototype cells and their influence fields. The recognition of hand postures is based on the topological features of the hand that are extracted from the binary image of the segmented hand region. The topological

References (21)

P. Dario et al.
Robot assistants: applications and evolution
Robotics and Autonomous Systems
(1996)
M. Ejiri
Towards meaningful robotics for the future: are we headed in the right direction
Robotics and Autonomous Systems
(1996)
K. Kawamura et al.
Design philosophy for service robots
Robotics and Autonomous Systems
(1996)
X. Yin et al.
Hand image segmentation using color and RCE neural network
International journal of Robotics and Autonomous System
(2001)
G.A. Bekey
Needs for robotics in emerging application: a research agenda
IEEE Transaction on Robotics and Automation
(1997)
G. Kaplan
Technology 1998 analysis and forecast: industrial electronics
IEEE Spectrum
(1998)
M.W. Krueger
Artificial Reality II
(1991)
J. Triesch, C.V.D. Malsburg, A gesture interface for human–robot interaction, in: Proceedings of 3rd IEEE International...
R. Kjeldsen, J. Kender, Toward the use of gesture in traditional user interfaces, in: Proceedings of International...
R.E. Kahn, M.J. Swain, P.N. Prokopowicz, R.J. Firby, Gesture recognition using the perseus architecture, in:...

There are more references available in the full text version of this article.

Cited by (54)

Thermal heat distribution features for hand identification
2022, Expert Systems with Applications
Citation Excerpt :
Analysis of palmprint images by their texture structure using the Fourier transform was proposed in Yang (2011). Other approaches that are based on palm–finger segmentation and data fusion, account for hand and finger motion by decomposing the hand silhouette in different regions corresponding to the back of the palm and the fingers (Yin & Xie, 2007). Several recent papers extract geometric features, like finger lengths, area, perimeter, solidity from regular images (Bera & Bhattacharjee, 2017; Luque-Baena et al., 2013) and from thermal images (Barra et al., 2019).
Thermal imaging is a non-invasive and portable technique with growing use in medical and authentication applications. This research utilizes thermal images for hand identification. Existing hand identification methods mainly extract geometric features, such as the palm’s and fingers’ absolute sizes and ratios. In this work, subject identification based on spatial statistical thermal distribution features is examined. These features do not depend on the geometric shape of the hand, but rather capture only physiological properties and convert them to statistical features. Thus, our goal is to evaluate the ability to identify a hand by characterizing the thermal heat distribution pattern, without relaying on geometric proprieties.
A novel image processing algorithm, which identifies and locates the hand posture from the image and extracts several features from 15 locations in the hand, was developed. Then, dimensionality reduction methods are applied to form a compact model of the data. The best model was selected based on the clustering quality. Classification of a single subject out of many resulted with accuracy of 94%. Classification of many subjects simultaneously resulted with accuracy of 91%. Identification of new image data is carried out in this low-dimensional space and results with an accuracy of 94%. In addition, the low-dimensional representation was used to explore time and environmental factors, which were found to have a low impact on the coded data, thus promoted the suitability of the proposed method.
Gesture recognition system for real-time mobile robot control based on inertial sensors and motion strings
2017, Engineering Applications of Artificial Intelligence
Citation Excerpt :
As an example, forearm electromyography (EMG) has been shown to provide accurate representations of hand movements for robot control (Wolf et al., 2013). In research related to human–robot interaction, successful application of hand gestures to control robotic service tasks (Fujii et al., 2014; Yin and Xie, 2007), robot manipulators (Khassanov et al., 2014; Wang et al., 2013), or humanoid robots (Riek et al., 2010) has been reported. A cooperative surgical robot system, guided by hand gestures and supported by an augmented reality, has been successfully implemented in Wen et al. (2014).
Navigating and controlling a mobile robot in an indoor or outdoor environment by using a range of body-worn sensors is becoming an increasingly interesting research area in the robotics community. In such scenarios, hand gestures offer some unique capabilities for human–robot interaction inherent to nonverbal communication with features and application scenarios not possible with the currently predominant vision-based systems. Therefore, in this paper, we propose and develop an effective inertial-sensor-based system, worn by the user, along with a microprocessor and wireless module for communication with the robot at distances of up to 250 m. Possible features describing hand-gesture dynamics are introduced and their feasibility is demonstrated in an off-line scenario by using several classification methods (e.g., random forests and artificial neural networks). Refined motion features are then used in K-means unsupervised clustering for motion primitive extraction, which forms the motion strings used for real-time classification. The system demonstrated an $F 1$ score of $90.05 %$ with the possibility of gesture spotting and null class classification (e.g., undefined gestures were discarded from the analysis). Finally, to demonstrate the feasibility of the proposed algorithm, it was implemented in an Arduino-based $8$ -bit ATmega2560 microcontroller for control of a mobile, tracked robot platform.
Signer-independence finger alphabet recognition using discrete wavelet transform and area level run lengths
2016, Journal of Visual Communication and Image Representation
Citation Excerpt :
In this paper, a sign with fingers located on the top of the palm is called a vertical sign, and a sign with fingers that are mainly located on the left side of the palm is called a horizontal sign, as shown in Fig. 21(a) and (b), respectively. Previous studies have usually recognized non-fist signs based on the curvature method and geometric computation [50,51]. Both the curvature method and geometric computation are used to find the number of extended fingers, using simple steps.
This paper proposes a method for finger alphabet recognition from backhand images with signer-independence. Input images that are divided into fist sign and non-fist sign groups should be analyzed and processed in different ways. Finger alphabets in the fist group are represented by a one-dimensional signal that represents the external hand boundaries. Its low and high frequency components are then extracted by discrete wavelet transform, which are key features for recognition. The non-fist sign images, which are radically digitized into a 20 × 20 block mask in terms of the hand geometry, due to the hand’s physical structure, can be recognized by the patterns of the occupied blocks. The experimental results show that the proposed method has a high likelihood of differentiating twenty-three static finger alphabets of backhand images. The proposed method reaches an improvement of 27.86% in recognition accuracy on a significant dataset of fist signs that includes multiple users, while the statistical distribution of the area level run length algorithm outperforms previous forehand approaches by 89.38% in recognition accuracy.
Comparative Analysis of Human Hand Gesture Recognition in Real-Time Healthcare Applications
2023, Lecture Notes in Networks and Systems
Approach to hand posture recognition based on hand shape features for human–robot interaction
2022, Complex and Intelligent Systems
MyoSpring: 3D Printing Mechanomyographic Sensors for Subtle Finger Gesture Recognition
2022, ACM International Conference Proceeding Series

View all citing articles on Scopus

View full text

Finger identification and hand posture recognition for human–robot interaction

Abstract

Introduction

Section snippets

Hand image segmentation

Feature selection

Humanoid service robot

Conclusions

Robotics and Autonomous Systems

Robotics and Autonomous Systems

Robotics and Autonomous Systems

International journal of Robotics and Autonomous System

Needs for robotics in emerging application: a research agenda

IEEE Transaction on Robotics and Automation

Technology 1998 analysis and forecast: industrial electronics

IEEE Spectrum

Artificial Reality II