Human tracking from a mobile agent: Optical flow and Kalman filter arbitration

doi:10.1016/j.image.2011.06.005

Signal Processing: Image Communication

Volume 27, Issue 1, January 2012, Pages 83-95

https://doi.org/10.1016/j.image.2011.06.005 Get rights and content

Abstract

Tracking moving objects is one of the most important but problematic features of motion analysis and understanding. The Kalman filter (KF) has commonly been used for estimation and prediction of the target position in succeeding frames. In this paper, we propose a novel and efficient method of tracking, which performs well even when the target takes a sudden turn during its motion. The proposed method arbitrates between KF and Optical flow (OF) to improve the tracking performance. Our system utilizes a laser to measure the distance to the nearest obstacle and an infrared camera to find the target. The relative data is then fused with the Arbitrate OFKF filter to perform real-time tracking. Experimental results show our suggested approach is very effective and reliable for estimating and tracking moving objects.

Highlights

► An autonomous mobile robot is designed to track a human using an infrared camera. ► The proposed method arbitrates between Kalman filter and Optical flow to improve the tracking performance. ► The algorithm has been successfully tested in real-time tracking of a man in an indoor lab environment. ► Compare the results to other approaches such as particle filters.

Introduction

Human tracking using mobile robots serves a lot of attention because an automated system estimating and tracking moving objects has many potential applications in the field of surveillance and engineering [46]. The mobile robot target tracking mechanism essentially needs the motion analysis of the target behavior. A number of approaches on prediction and tracking are based on the traditional Kalman filter (KF) [10], [11], [12], [13], [14], [15]. In the KF approach, it is presumed that the behavior of a moving target could be characterized by a predefined model, and the models can be represented in terms of a state vector. In reality, however, these models fail to characterize the motion of moving targets accurately. As a result of this, KF fails to track a target, especially when there are occlusions caused by other objects or sudden changes in the trajectory of the motion [13]. In this paper, we propose an efficient method of tracking the target called Arbitrate OFKF, which will overcome the abovementioned problem and work successfully even when sudden changes in trajectory occur. This proposed method arbitrates the output of OF and KF depending on the trajectory of the previous motion. Our arbitration algorithm is such that the output, during sharp turns, will be taken from OF algorithm, and, in all other instances, from KF algorithm.

There are some visual restrictions in which any methods cannot be applicable, for instance in industrial applications, like the ones examined in visual environments and applications [48], airports with crowded conditions, industrial process monitoring [49] and environmental monitoring. We have exploited an infra-camera for visual content, in which the presented technique is applicable as other types of visual contents that can potentially solve these existing issues in [48], [49].

The paper is organized as follows. Section 2 introduces the related work in the tracking using a mobile robot. Section 3 explains the piecewise constant acceleration model for implementation of KF. In Section 4, a solution for improving KF based tracking using arbitration between KF and OF. Section 5 reports experimental results and considerations. Finally, conclusions and future work are illustrated in Section 6.

Section snippets

Mobile robot

Tracking people by the use of a mobile robot is an essential task for the coming generation of service and human-interaction robots. In the field of computer vision, the tracking of moving objects by mobile robot is a relatively hard problem. The system described in [1] uses a sample-based joint probabilistic data association filter for human tracking with a mobile robot. A fuzzy logic approach is used for navigation of the mobile robot [2] whereas the neurofuzzy-based approach for tracking

Traditional methods: Framing the model of target tracking

First, we choose a state vector to contain information about the position and the velocity of the man. We then update the state according to our model called the constant acceleration model, in which the value of acceleration is calculated from the last few states, using a Taylor series expansion of the present expansion. The state can thus be thought of as being updated through the equations: $x_{k + 1} = A x_{k} + B \frac{d x_{k}}{d t_{k}} + w_{k}$

And the output equations may be written as $y_{k} = C x_{k} + z_{k},$ where A, B and C are matrices of

Proposed pan-tilt operation

In the target region of interest (ROI) tracking, we need to find the required pan and tilt angle for the specific camera configuration as shown in Fig. 2 so that the camera head can rotate to track the target. In other words, this is the angle vector θ required to make image center coincident with the target centroid: $θ = [\begin{matrix} θ_{x} \\ θ_{y} \end{matrix}],$ where the rotational angle (θ_x) with respect of the center of the image is the only angle associated with pan defined by ΔX. It is similar with (θ_y), when there is only

Arbitration of Optical flow and Kalman filter

We have applied the Optical flow as shown in the flow chart in Fig. 4 for the tracking process. We further propose the Arbitrate OF and KF algorithm to calculate the estimated value of the pan angle, which will be used for our prediction. In our proposed method, both KF and OF will predict the value of the pan angle but, depending on the situation, we will decide which value should be fed to the robot for tracking the target. If the man walks smoothly, then the pan angle value predicted by KF

Experimental results

This section compares the results obtained from algorithms using only KF, only OF, and finally the Arbitrate OFKF model developed in this paper. Since we have tracked a single point in the KF, we have used only one feature point in the OF for comparisons. A final comparison is made between Arbitrate OFKF and the particle filter at the end of the section. In the series of the experiments, we have specifically evaluated pan angle values.

Conclusion

In this paper, a novel tracking method is proposed, which is able to predict a target position very efficiently even if the target object turns suddenly during its motion. The proposed method is based on an arbitration between OF and KF. It takes into consideration the trajectory of the target motion, and gives a much better result of tracking than individual OF or KF filters. In this paper, attention has been drawn to different scenarios where either the KF works better or the OF does. A

Acknowledgment

This study was supported in part by the School of Engineering at Virginia Commonwealth University (VCU) through the internship program between Indian Institute of Technology, Kharagpur and VCU. The first author also acknowledges the support of NSF Division of Electrical, Communications and Cyber Systems, CAREER Award #1054333.

References (49)

E. Aguirre et al.
Fuzzy behaviors for mobile robot navigation: design, coordination and fusion
International Journal of Approximate Reasoning
(2000)
J.B. Mbede et al.
Intelligent mobile manipulator navigation using adaptive neuro-fuzzy systems
Information Sciences
(2005)
D.S. Jang et al.
2D human body tracking with structural Kalman filter
Pattern Recognition
(2002)
S. Jung et al.
Tracking and motion estimation of the articulated object: a hierarchical Kalman filter approach
Real-Time Imaging
(1997)
R. Rosales et al.
A framework for heading-guided recognition of human activity
Computer Vision and Image Understanding
(2003)
S.G. Wu et al.
Hand tracking in a natural conversational environment by the interacting multiple model and probabilistic data association (IMM-PDA) algorithm
Pattern Recognition
(2005)
B. Horn et al.
Determining optical flow
Artificial Intelligence
(1981)
J.S. Park et al.
Contour matching: a curvature-based approach
Image and Vision Computing
(1998)
K. Jung
Neural network-based text location in color images
Pattern Recognition Letters
(2001)
T. Kampke
Functional arbitration for multipurpose senso-motory systems
Robotics and Autonomous Systems
(1999)

Y. Motai

Visual-based human–robotic interaction for extracting salient features of an industrial object for an automated assembly system, Special Issue on Machine Vision

International Journal of Computers in Industry

(2005)

W. Schulz et al.

People tracking with mobile robots using sample-based joint probabilistic data association filters

International Journal of Robotics Research

(2003)

A.M. Zhu et al.

Neurofuzzy-based approach to mobile robot navigation in unknown environments

IEEE Transactions on Systems Man and Cybernetics Part C—Applications and Reviews

(2007)

S.J. Kim, W.K. Choi, H.T. Jeon, Intelligent robot control with personal digital assistants using fuzzy logic and neural...

J.B. Mbede et al.

Fuzzy and recurrent neural network motion control among dynamic obstacles for robot manipulators

Journal of Intelligent & Robotic Systems

(2001)

E. Rimon et al.

Exact robot navigation using artificial potential functions

IEEE Transactions on Robotics and Automation

(1992)

J. Borenstein et al.

The vector field histogram—fast obstacle aviodance for mobile robots

IEEE Transactions on Robotics and Automation

(1991)

R. Polana, R. Nelson, Low level recognition of human motion (or how to get your man without finding his body parts),...

M. Mikic et al.

Human body model acquisition and tracking using voxel data

International Journal of Computer Vision

(2003)

A.K. Rastogi et al.

Design of a real-time tracking system for fast-moving objects

IETE Journal of Research

(1997)

X.D. Sun et al.

Region of interest extraction and virtual camera control based on panoramic video capturing

IEEE Transactions on Multimedia

(2005)

X.P. Yun et al.

Design, implementation, and experimental results of a quaternion-based Kalman filter for human body motion tracking

IEEE Transactions on Robotics

(2006)

Beymer,K. Konolige, Tracking people from a mobile platform, in: IJCAI-2001 Workshop on Reasoning with Uncertainty in...

D. Schulz et al.

People tracking with mobile robots using sample-based joint probabilistic data association filters

International Journal of Robotics Research

(2003)

Cited by (50)

A real-time, continuous pedestrian tracking and positioning method with multiple coordinated overhead-view cameras
2021, Measurement: Journal of the International Measurement Confederation
This paper proposes a real-time, continuous pedestrian tracking and positioning method with multiple coordinated overhead-view cameras. First, a discriminative correlation filter with detection and space reliability (DSR-DCF) is presented for continuous, real-time pedestrian tracking with single cameras. Then, the frame difference method is employed to detect moving pedestrians when they walk from one camera’s view to another’s. Motion-area images are cropped to establish a pedestrian image library (PIL). Finally, this PIL is relied upon to reidentify pedestrians. Pedestrian positioning is realized based on the pinhole imaging model. Tracking and positioning experiments showed the following. DSR-DCF is stable with a high success rate. Its tracking speed exceeds 24 fps. It produces a success rate higher than 90% for cross-camera pedestrian tracking and misses no pedestrians. It yields stable pedestrian positioning results with an accuracy better than 0.2 m. The proposed algorithm will be used in airport terminals.
A non-local propagation filtering scheme for edge-preserving in variational optical flow computation
2021, Signal Processing: Image Communication
The median filtering heuristic is considered to be an indispensable tool for the currently popular variational optical flow computation. Its attractive advantages are that outliers reduction is attained while image edges and motion boundaries are preserved. However, it still may generate blurring at image edges and motion boundaries caused by large displacement, motion occlusion, complex texture, and illumination change. In this paper, we present a non-local propagation filtering scheme to deal with the above problem during the coarse-to-fine optical flow computation. First, we analyze the connection between the weighted median filtering and the blurring of image edge and motion boundary under the coarse-to-fine optical flow computing scheme. Second, to improve the quality of the initial flow field, we introduce a non-local propagation filter to reduce outliers while preserving context information of the flow field. Furthermore, we present an optimization combination of non-local propagation filtering and weighted median filtering for the flow field estimation under the coarse-to-fine scheme. Extensive experiments on public optical flow benchmarks demonstrate that the proposed scheme can effectively improve the accuracy and robustness of optical flow estimation.
CNN and HOG based comparison study for complete occlusion handling in human tracking
2020, Measurement: Journal of the International Measurement Confederation
Citation Excerpt :
It was aimed to distinguish pose change resulting from partial or complete occlusion. In a different study by Motai, et al. [45], KF and Optical Flow (OF) were used together for sudden maneuverable human tracking. Laser and infrared cameras were used for that study.
Applications in the Human Machine Interaction (HMI) field are increasing since people and robots share the same environment and perform activities together. Therefore, robust detection and strong tracking methods are required. This study is about human detection and tracking. For detection, two different approaches, Histogram of Oriented Gradients (HOG)-Support Vector Machine (SVM) and Convolutional Neural Network (CNN) are used. The video used in the application is recorded in a noisy environment, and in some frames, the human is completely covered by other objects in the environment, i.e. exposed to complete occlusion. For this reason, Kalman Filter (KF) and Particle Filter (PF) based tracking is performed. Different from these two traditional filters, a hybrid Kalman-Particle Filter (KPF) has also been proposed. The contribution of this study is to compare CNN with HOG-SVM, which is described as the most successful human detection method. As a result of the study, it was found that CNN provides successful results especially in case of occlusion. This study is also important in terms of proposing a hybrid KPF and comparing the filters in the case of complete occlusion. The results showed that for human tracking, CNN using KF performed better performance throughout the video. The proposed KPF also outperformed PF and this superiority became much more prominent in the case of complete occlusion.
Re-projected SURF Features Based Mean-Shift Algorithm for Visual Tracking
2020, Procedia Computer Science
Visual tracking is an art of tracking a moving object over video frames using non-stationary cameras, for which feature descriptors of the target are computed and applied to a motion model. In this paper, SURF is used to compute the feature descriptors and Mean-Shift algorithm as a motion model. Mean-shift algorithm is fundamentally a logical approach to track the object on an image frame where the appearance is described by histograms. Mean-Shift tracking can directly be applied to SURF features but there is a big constraint of availability of an adequate number of feature keypoints for a given object. To get over this limitation, the re-projection technique is implemented in this paper for tracking an object in any video recorded from a mobile or stationary camera. Re-projection, online updates the histogram of object template for every frame by superimposing the feature keypoints from upcoming frames to the first frame. The object can be tracked using SURF descriptors without using any secondary information (color, texture, optical flow, gradient, etc.) regarding the object, which makes it computationally inexpensive to be used in real-time systems. Also, the SURF feature is calculated only for the target object that is to be tracked, this further reduces computational complexity. The simulation results prove the effectiveness of the work.
A novel fusion framework for robust human tracking by a service robot
2017, Robotics and Autonomous Systems
Citation Excerpt :
Additionally, local feature descriptors are proven to be a good choice for image matching tasks on a mobile platform, where occlusions and missing objects can be handled [8,9]. These robust local feature descriptors are being increasingly used for visual object tracking applications [10,11,6,7]. Compared to SIFT and SURF which are computationally heavy, a number of light weight binary point features like BRIEF [12], BRISK [13], ORB [14], FREAK [15] and DAISY [16] have been discovered that are being used for applications such as place recognition in visual SLAM [17].
This work investigates the problem of robust vision-based human tracking by a human following robot using point-based features like SURF. The problem is challenging owing to failures arising because of variation in illumination, change in pose, size or scale, camera motion and partial or full occlusion. While point-based features provide robust detection against photometric and geometric distortions, the tracking of these features over subsequent frames becomes difficult as the number of matching points between a pair of images drops quickly with slight variation in target attribute owing to above mentioned variations. The problem of robust human tracking by the robot is solved by proposing a multi-tracker fusion framework that allows one to combine multiple tracker to ensure long term tracking of the target. This fusion framework also allows for creating a dynamic template pool of target features which gets updated over time. The interaction between the first two trackers is used to update the template pool of the target attribute while the last tracker is used to estimate the location of the target in case of full occlusion. The working of the framework is demonstrated by combining a SURF-based mean-shift tracker, an optical-flow tracker and a Kalman filter to provide robust tracking over a long time. The efficacy of the resulting tracker is demonstrated through rigorous testing on a variety of video datasets.
A fractional order variational model for the robust estimation of optical flow from image sequences
2016, Optik
In this paper, a fractional order variational model for estimating the optical flow is presented. In particular, the proposed model generalizes the integer order variational optical flow models. The fractional order derivative describes discontinuous information about texture and edges, and therefore a more suitable in estimating the optical flow. The proposed variational functional is a combination of a global model of Horn and Schunck and the classical model of Nagel and Enkelmann. This formulation yields a dense flow and preserves discontinuities in the flow field and also provides a significant robustness against outliers. The Grünwald–Letnikov derivative is used for solving complex fractional order partial differential equations. The corresponding linear system of equations is solved by an efficient numerical scheme. A detailed stability and convergence analysis is given in order to show the mathematical applicability of the numerical algorithm. Experimental results on various datasets verify the validity of the proposed model.

View all citing articles on Scopus

View full text

Human tracking from a mobile agent: Optical flow and Kalman filter arbitration

Abstract

Highlights

Introduction

Section snippets

Mobile robot

Traditional methods: Framing the model of target tracking

Proposed pan-tilt operation

Arbitration of Optical flow and Kalman filter

Experimental results

Conclusion

Acknowledgment

International Journal of Approximate Reasoning

Information Sciences

Pattern Recognition

Real-Time Imaging

Computer Vision and Image Understanding

Pattern Recognition

Artificial Intelligence

Image and Vision Computing

Pattern Recognition Letters

Robotics and Autonomous Systems

International Journal of Computers in Industry

People tracking with mobile robots using sample-based joint probabilistic data association filters

International Journal of Robotics Research

Neurofuzzy-based approach to mobile robot navigation in unknown environments

IEEE Transactions on Systems Man and Cybernetics Part C—Applications and Reviews

Fuzzy and recurrent neural network motion control among dynamic obstacles for robot manipulators

Journal of Intelligent & Robotic Systems

Exact robot navigation using artificial potential functions

IEEE Transactions on Robotics and Automation

The vector field histogram—fast obstacle aviodance for mobile robots

IEEE Transactions on Robotics and Automation

Human body model acquisition and tracking using voxel data

International Journal of Computer Vision

Design of a real-time tracking system for fast-moving objects

IETE Journal of Research

Region of interest extraction and virtual camera control based on panoramic video capturing

IEEE Transactions on Multimedia

Design, implementation, and experimental results of a quaternion-based Kalman filter for human body motion tracking

IEEE Transactions on Robotics

People tracking with mobile robots using sample-based joint probabilistic data association filters

International Journal of Robotics Research