Elsevier

Pattern Recognition

Volume 48, Issue 8, August 2015, Pages 2407-2417
Pattern Recognition

Modeling and recognizing human trajectories with beta process hidden Markov models

https://doi.org/10.1016/j.patcog.2015.02.028Get rights and content

Highlights

  • We propose a new method for human trajectory modeling.

  • Both the number of motions and the sharing characteristic are inferred from data.

  • An efficient MCMC algorithm is developed.

  • Experimental results show the superiority of our approach over others.

Abstract

Trajectory-based human activity recognition aims at understanding human behaviors in video sequences, which is important for intelligent surveillance. Some existing approaches to this problem, e.g., the hierarchical Dirichlet process hidden Markov models (HDP-HMM), have a severe limitation, namely the motions are shared among trajectories from the same activity and not shared among activities (classes). To overcome this shortcoming, we propose a new method for modeling human trajectories based on the beta process hidden Markov models (BP-HMM) where the motions are selectively shared among trajectories. All the trajectories from different activities can be jointly modeled with a BP-HMM, which allows motions being shared among activities. Using our technique, the number of available motions and the sharing patterns can be inferred automatically from training data. We develop an efficient Markov chain Monte Carlo algorithm for model training. Experiments on both synthetic and real data sets demonstrate the effectiveness of our approach.

Introduction

Effective human activity recognition (HAR) is crucial for the successful application of intelligent surveillance systems, which has drawn growing interest during recent years. The purpose of HAR is to understand what people are doing from their position [1], figure [2], motion [3], or other spatio-temporal information derived from video sequences. Research in this field mainly includes three stages: (a) object segmentation and tracking, (b) feature extraction, and (c) activity classification, which are also typical steps of designing a surveillance system. In this paper, we focus on the activity classification tasks, or more specifically, learning to recognize human behaviors from trajectory data [4], [5], [6].

An example of this kind of problem can be found in a shopping mall. In this scene, customers take different classes of normal activities such as “entering the shop”, “leaving the shop”, “passing the shop”, etc. Meanwhile, some abnormal activities may also occur, such as “wandering” and “fighting”. In order to detect the abnormal behaviors, a surveillance system should have the ability to characterize and classify the activities. Due to a growing need from the public safety and security, this requirement is also prevalent in many other scenes, such as airports, railway stations, and banks.

From daily experience we know that a human activity can be modeled by transitions among simple motions, and moreover, multiple trajectories may share the same motions. For example, in a certain shopping mall, the activity of a customer “entering the shop” may be decomposed into “moving east” first and then “moving north”, and the activity of a customer “leaving the shop” may be decomposed into “moving south” first and then “moving east”. In this simple case, the motion “moving east” is shared while the motions “moving north” and “moving south” are class-specific.

Hidden Markov models (HMMs) are a popular class of models in this field, by which the invisible motions can be modeled as hidden states and the hidden state sequence corresponding to a human trajectory forms a Markov chain. Moreover, the transitions between the hidden states in an HMM represent the transitions between the motions from a human activity. Therefore, HMMs are capable to capture intrinsic structures of activities. However, a serious limitation of the standard HMMs is that one needs to define an appropriate number of hidden states. Models with too few states lack flexibility while too many states lead to the overfitting problem. Another challenge is that the standard HMMs do not have the capability to jointly model multiple activities with shared motions. Nevertheless, in the real HAR practice, motions are often shared among different activities. Eliminating these limitations can result in more flexible models, which is also our desideratum.

In this paper, we propose a method for trajectory-based HAR tasks. We first try to discover a set of latent motions, including the shared motions and the unique motions, in specific trajectories. Then we model the trajectories by transitions of different motions. A Markov chain Monte Carlo (MCMC) algorithm is developed for efficient model training. The final classifier for HAR tasks is given by maximizing the log-likelihood of a test trajectory. Our work is inspired by the beta process hidden Markov models (BP-HMM) [7], which is concerned with the patterns of sharing motions. In particular, motions shared among the multiple time series or unique can be inferred from data. This is quite different from other HMM-based approaches [1], [12], [13], [14], [15] where the state spaces, the transitions among these states and the emission parameters are exactly shared in multiple time series from one kind of activity. The characteristics of soft-sharing motions in BP-HMM help to jointly model all the trajectories from different activities with one model, which allows motions being shared among activities. In contrast, in other methods [1], [12], [13], [14], [15] trajectories from different activities are separately modeled with different parameters for classification, which causes that motions cannot be shared among activities. Important distinctions between (BP-HMM) [7] and our method are as follows. First, the main dynamical system in the BP-HMM is a vector autoregressive process while we use an HMM with Gaussian emissions which is more appropriate for trajectory-based HAR tasks. Second, our sampling method is partially different from the BP-HMM, which is more suitable for our model. Third, we employ BP-HMM on human trajectory recognition through jointly modeling all the trajectories for training and performing an additional procedure on trajectory-specific transitions before classification. A preliminary result was reported previously [6]. This paper substantially extends that work in terms of additional knowledge presentations including Dirichlet process, hierarchical Dirichlet process and beta process, more understandable model descriptions, additional detailed sampling algorithm, more adequate experiments and additional performance and complexity analysis.

The remainder of this paper is organized as follows. Section 2 gives the overview of related HAR work and statistical models. Section 3 presents our method for modeling human trajectories and Section 4 introduces the adopted parameter inference techniques. Section 5 gives a detailed algorithm with complexity analysis for sampling. After providing the classification rule in Section 6, Section 7 reports experimental results on both synthetic and real data sets including comparisons with other methods. Finally, Section 8 gives conclusions and future research directions.

Section snippets

Related HAR work and statistical models

HMMs are widely adopted methods for trajectory recognition, which model the temporal evolution of motions in a human trajectory as a Markov chain. The real challenge of these methods is to model the hidden states. Bashir et al. [8] considered the hidden states as subtrajectories. They segmented a trajectory at points of change in curvature and represented the subtrajectories by their principal component analysis coefficients. Nascimento et al. [1] considered the hidden states as motions to be

Modeling human activities with BP-HMM

Our task is to map a sequential trajectory x to a single activity label y. Formally, let x=(x1,x2,,xT) be a specific trajectory where xtR2 denotes the displacement of a person from time t1 to time t. Note that the two components of vector xt respectively correspond to vertical and horizontal displacement. zt denotes the invisible motion label of xt. In our model each xt is a draw from a Gaussian distribution with unknown mean and covariance:xt~N(μzt,Σzt).We place a conjugate

Training with a MCMC sampling method

Let X={(x(1),y1),(x(2),y2),,(x(N),yN)} be all of our training data, where each (x(i),yi) denotes a specific labeled trajectory. x(i) is the ith sequence of observations and yi is the corresponding activity labels represented by a set of constants. Assume we have Y activities, then yi{1,2,,Y}. Since the motions are globally shared while the transitions are trajectory-specific, the parameters to be learned are Π={π(1),,π(N)} and Θ={(μ1,Σ1),,(μK,ΣK)}, where each π(i)={π1(i),π2(i),,πK(i)}

Algorithm and complexity

In this section, we give a detailed MCMC sampling procedure for binary feature assignments f(i), state sequences z(i), transition distribution η(i) and dynamic parameters {μk,Σk} in order. Algorithm 1 describes one iteration of the sampling process, in which the complexity of each sub-process is analyzed. Note that Ki is the number of active motion states in sequence i and Ti is the number of timesteps.

Algorithm 1

Given ηold(i), fold(i), zold(i), {μk,Σk}old,
Set η(i)=ηold(i), f(i)=fold(i), z(i)=zold(i) and

Classification

In order to get the expected class-specific transitions {π1,,πY}, we perform an additional procedure on the trajectory-specific transitions {π(1),,π(N)} before classification:πy=iyi=yπ(i)Ny,where Ny is the number of training trajectories which belong to class y.

For testing, given a new trajectory x, we classify it into activity y{1,,Y} by maximizing the log-likelihood:y=argmaxyY{logp(x|πy,Θ},The observation likelihood p(x|πy,Θ) can be computed directly using a forward message passing

Experiments

We test the performance of our model on both synthetic and real data. The synthetic data have two classes of simple activities, which aims at demonstrating the capability of our approach to recover the true model. Experimental results on data from real-world scenes include comparisons with state-of-the-art methods.

Conclusion

In this paper, we have presented a method for modeling and recognizing human activities. We model the distributions of displacements in a human trajectory as Gaussians and the temporal evolution of invisible motions as a Markov chain. Using a beta process prior, our approach discovers a set of latent motions that are selectively shared among multiple trajectories and among different activities. Parameters are learned from an efficient MCMC sampling algorithm. For recognition, a test trajectory

Conflict of interest

None declared.

Acknowledgements

We are grateful to the anonymous reviewers for their helpful comments to improve the paper. This work is supported by the National Natural Science Foundation of China under Projects 61370175 and 61075005, and Shanghai Knowledge Service Platform Project (No. ZF1213).

Shiliang Sun received the B.E. degree in Automatic Control from Beijing University of Aeronautics and Astronautics and Ph.D. degree in Pattern Recognition and Intelligent Systems from Tsinghua University in 2002 and 2007, respectively. Now he is a professor and the director of the Pattern Recognition and Machine Learning Research Group, Department of Computer Science and Technology, East China Normal University. He is on the editorial boards of several international journals and a referee for

References (20)

  • J.C. Nascimento et al.

    Trajectory classification using switched dynamical hidden Markov models

    IEEE Trans. Image Process.

    (2010)
  • N. Vaswani et al.

    “Shape activity”a continuous-state HMM for moving/deforming shapes with application to abnormal activity detection

    IEEE Trans. Image Process.

    (2005)
  • C. Sminchisescu, A. Kanaujia, Z. Li, D. Metaxas, Conditional models for contextual human motion recognition, in:...
  • Q. Gao, S. Sun, Trajectory-based human activity recognition using hidden conditional random fields, in: Proceedings of...
  • Q. Gao, S. Sun, Trajectory-based human activity recognition with hierarchical Dirichlet process hidden Markov models,...
  • Q. Gao, S. Sun, Human activity recognition with beta process hidden Markov models, in: Proceedings of the International...
  • E.B. Fox, E.B. Sudderth, M.I. Jordan, A.S. Willsky, Joint modeling of multiple related time series via the beta...
  • F. Bashir et al.

    Object trajectory-based activity classification and recognition using hidden Markov models

    IEEE Trans. Image Process.

    (2007)
  • L. Liao et al.

    Hierarchical conditional random fields for GPS-based activity recognition

    Springer Tracts Adv. Robot.

    (2007)
  • D. Vail, M.M. Veloso, J. Lafferty, Conditional random fields for activity recognition, in: Proceedings of the 6th...
There are more references available in the full text version of this article.

Cited by (21)

  • Dynamic temporal ADS-B data attack detection based on sHDP-HMM

    2020, Computers and Security
    Citation Excerpt :

    But the multiple attribute data is not analysed fully and hyper-parameters are not configured dynamically. The sHDP-HMM is good at analysing and predicting time series data (Sun et al., 2015). For the model, the hidden states can be used to classify the sequential data and set up the prediction foundation.

  • Semantic trajectory insights for worker safety in dynamic environments

    2019, Automation in Construction
    Citation Excerpt :

    From our knowledge, no previous work was done to understand workers' mobility patterns within the stay regions using HMMs on semantic trajectories. After reviewing the most relevant literature [74–79] on the understanding users' behaviors in dynamic environments, using the scenario (see Fig. 4) a prototype system named ‘WoTAS’ is constructed by considering the roles of building supervisors and H&S managers. The system is developed and applied to an already constructed building.

  • A review on classifying abnormal behavior in crowd scene

    2019, Journal of Visual Communication and Image Representation
    Citation Excerpt :

    The implementation of each model which being implies for the features of trajectory, motion and motion-HOG are explained extensively. Hidden Markov Model (HMM) is a dynamic probabilistic method utilized in countless area such as traffic surveillance [61,62], trajectories analysis [63,64] and recognizing human action [65,66]. In general, HMM is a double stochastic process consisting of hidden and observes processes.

  • Naïve Bayes switching linear dynamical system: A model for dynamic system modelling, classification, and information fusion

    2018, Information Fusion
    Citation Excerpt :

    By modelling motion patterns, the model is able to track pedestrians under long term occlusions. Sun et al. [19] present a method for modelling and recognising human trajectories using a beta process HMM. The beta process approach allows for all trajectories of different activities to be jointly modelled.

  • Hierarchically linked infinite hidden Markov model based trajectory analysis and semantic region retrieval in a trajectory dataset

    2017, Expert Systems with Applications
    Citation Excerpt :

    In video analysis, one of the well-known approach is to exploit HMMs. HMMs are useful for modeling invisible motions and latent structures in a given scene (Sun, Zhao, & Gao, 2015). Many literatures exploited HMMs on diverse applications.

View all citing articles on Scopus

Shiliang Sun received the B.E. degree in Automatic Control from Beijing University of Aeronautics and Astronautics and Ph.D. degree in Pattern Recognition and Intelligent Systems from Tsinghua University in 2002 and 2007, respectively. Now he is a professor and the director of the Pattern Recognition and Machine Learning Research Group, Department of Computer Science and Technology, East China Normal University. He is on the editorial boards of several international journals and a referee for many top journals. His research interests include machine learning, pattern recognition, computer vision and natural language processing, etc.

Jing Zhao received the B.E. degree in Computer Science and Technology from East China Normal University in 2011. Now she is a Ph.D. student in the Pattern Recognition and Machine Learning Research Group, Department of Computer Science and Technology, East China Normal University. Her research interests include machine learning, pattern recognition, etc.

Qingbin Gao received the B.E. degree in Computer Science and Technology from East China Normal University in 2010. Now he is a master student in the Pattern Recognition and Machine Learning Research Group, Department of Computer Science and Technology, East China Normal University. His research interests include machine learning, pattern recognition, etc.

View full text