Skeleton-based traffic command recognition at road intersections for intelligent vehicles

Highlights

• Pioneering research on traffic command recognition distinguishing directions and gestures.

• A two-stage recognition model exploiting skeletal geometry and co-occurrence features.

• A specialized dataset for recognizing Chinese traffic commands at road intersections.

Abstract

Understanding traffic officer commands is a fundamental perception task for intelligent vehicles in driver assistance and autonomous driving. Previous studies have emphasized explicit traffic command gesture recognition but have not considered situations where the traffic officer is controlling the subjects in other directions, which would also influence decision-making of the ego vehicle. To fill in the gap, this article aims to research visual skeleton-based recognition of traffic commands occurring at road intersections, where both command directions and gestures should be determined. Specifically, a two-stage recognition framework for four cross-shaped directions and eight command gestures is proposed. Two kinds of handcrafted features, including upper-body geometric features and keypoint co-occurrence features, are established with estimated 2D human keypoint coordinates and heatmaps and further combined into a deep learning network. The first stage handles human body orientation classification, while the second stage addresses command gesture recognition with extra usage of the output from the first stage. Combining the recognized body orientation and command gesture, the type of traffic command can ultimately be inferred. For training and validation, a dataset termed the Chinese Traffic Command at Intersections (CTCX) is built. The proposed method gains an outperforming edit accuracy of 89.67% on the CTCX test set, demonstrating its effectiveness. This work provides a foundation in this area and is expected to inspire more research on traffic command recognition with directions in the near future.

Introduction

Just as human drivers can understand regulated traffic command gestures when traffic officers are controlling, intelligent vehicles should be capable of recognizing them as well. Traffic command gesture recognition is a fundamental perception task in driver assistance and autonomous driving. This task is particularly critical under mixed traffic scenarios because it can inform drivers or vehicles of driving situations and improve safety.

Recent years have witnessed a revolution in methodology and database research on traffic command gesture recognition [1], [2], [3], [4], [5], owing to the advancement of deep learning in computer vision, especially in human action recognition. Being aware of the significance of automatically recognizing traffic commands for intelligent driving, a growing number of researchers have devoted attention to related studies. To date, the proposed methods and public databases have been increasing, and the gap between their scientific research and real applications has been narrowing. Previous studies mainly aimed to recognize explicit commands, such as eight kinds of Chinese regulated traffic command gestures. Generally, only the command gestures directed to the vehicle must be recognized. However, command gestures in other directions also influence the ego vehicle. For example, as shown in Fig. 1, when a driver notices that a traffic officer gestures for the vehicles coming in the crosswise direction to go straight, he/she is obligated to stop at the same time. Humans are able to decipher these seemly unrelated gestures. Supposing a vehicle could be informed of the controls to other vehicles, it would have a more comprehensive knowledge of the surrounding environment and tend to make correct decisions rather than continue driving by mistake. Therefore, it is necessary to make intelligent vehicles learn these associated skills.

To serve the understanding of implicit traffic commands, we specify an extended traffic command recognition task to obtain awareness of both command gestures and directions. As a startup, direction recognition is simplified as a four-direction classification task, which adapts to the circumstance of typical road intersections. High accuracy and great robustness are the primary objectives of this new task for the sake of on-board practical applications. To our knowledge, this work is the first to clearly describe the issue and aim to resolve it.

The human skeleton is a compact representation for action and has been a widely used modality in human action recognition [6], [7], traffic agent intent [8] and trajectory prediction [9], [10]. Previous works on traffic command gesture recognition have also illustrated the superiority of skeleton information usage [4], [11], [5], [12]. Considering the compactness, robustness and reusability of skeleton modality, we propose a two-stage recognition framework based on an estimated 2D skeleton. The discriminative features are generated by combining the handcrafted skeletal geometry and co-occurrence with deep learning. The architecture of the two stages is simple, and the output of the first stage continues to be fed into the second stage as a part of the features. We also build a dataset containing Chinese traffic command gesture instances in the four directions for training and validation. To summarize, this work makes contributions in three ways:

• A pioneering study on traffic command recognition distinguishing directions and gestures is carried out to fill in research blanks, promoting the perception capability of intelligent vehicles.

• With estimated 2D human skeletons, a two-stage recognition framework with discriminative features combined with handcrafted skeletal geometry and co-occurrence is presented to tackle the challenge. Our approach is characterized by a simple but effective structure and attains significant performance in the experiments.

• An extended dataset termed “Chinese Traffic Command at Intersections” (CTCX) is established with videos of traffic command gestures in the four cross-shape directions for methodology studies. The experimental results verify the effectiveness of the proposed approach¹.

The rest of the article is organized as follows. Section 2 introduces the related and illuminating works. Section 3 gives a specific definition of the proposed problem, and Section 4 explains the details of the presented approach. Section 5 illustrates the dataset establishment and the evaluation metrics, demonstrates the experimental results and comparisons, and discusses the influences of model settings and parameter selection, as well as generalization. Finally, conclusions are drawn in Section 6.