Abstract
In this paper, we propose a hierarchical visual event pattern mining approach and utilize the patterns to address the key problems in video mining and understanding field. We classify events into primitive events (PEs) and compound events (CEs), where PEs are the units of CEs, and CEs serve as smooth priors and rules for PEs. We first propose a tensor-based video representation and Joint Matrix Factorization (JMF) for unsupervised primitive event categorization. Then we apply frequent pattern mining techniques to discover compound event pattern structures. After that, we utilize the two kinds of event patterns to address the applications of event recognition and anomaly detection. First we extend the Sequential Monte Carlo (SMC) method to recognition of live, sequential visual events. To accomplish this task we present a scheme that alternatively recognizes primitive and compound events in one framework. Then, we categorize the anomalies into abnormal events (never seen events) and abnormal contexts (rule breakers), and the two kinds of anomalies are detected simultaneously by embedding a deviation criterion into the SMC framework. Extensive experiments have been conducted which demonstrate that the proposed approach is effective as compared to other major approaches.
References
Blank B, Gorelick L, Shechtman E, Irani M, Basri R (2005) Actions as space-time shapes. Internatinal conference on computer vision
Boiman O, Irani M (2005) Detecting irregularities in images and in video. Internatinal conference on computer vision
Chan MT, Hoogs A, Schmiederer J, Petersen M (2004) Detecting rare events in video using semantic primitives with HMM. International conference on pattern recognition
Chris D, Li T, Peng W, Park H (2006) Orthogonal nonnegative matrix tri-factorizations for clustering. International conference on knowledge discovery and data mining
Cui P, Sun LF, Liu ZQ, Yang SQ (2007) A sequential monte carlo approach to anomaly detection in tracking visual events. International conference on computer vision and pattern recognition
Davis JW, Taylor SR (2002) Analysis and recognition of walking movements. International conference on pattern recognition
Doucet A, Freitas N, Gordon N (2001) Sequential monte carlo methods in practice. Springer, New York
Du YT, Chen F, Xu WL, Li YB (2006) Recognizing interaction activities using dynamic bayesian network. International conference on pattern recognition
Duong TV, Bui HH, Phung DQ, Venkatesh S (2005) Activity recognition and abnormality Detection with the switching hidden semi-markov model. International conference on computer vision and pattern recognition
Efros AA, Berg AC, Mori G, Malik J (2003) Recognizing action at a distance. International conference on computer vision
Gilbert A, Illingworth J, Bowden R (2009) Fast realistic multi-action recognition using mined dense spatio-temporal features. International conference on computer vision
Hakeem A, Shah M (2005) Multiple agent event detection and representation in videos. National conference on artificial intelligence
Haritaoglu I, Harwood D, Davis LS (2000) W4 real-time surveillance of people and their activities. IEEE Trans Pattern Anal Mach Intell 22(8): 809–830
Ivanov YA, Bobick AF (2000) Recognition of visual activities and interactions by stochastic parsing. IEEE Trans Pattern Anal Mach Intell 22: 852–872
Kawanaka D, Okatani T, Deguchi K (2006) HHMM based recognition of human activity. IEICE transactions on information and systems E89-D(7): 2180–2185
Ke Y, Sukthankar R, Hebert M (2005) Efficient visual event detection using volumetric features. International conference on computer vision
Li T, Chris D (2006) The relationships among various nonnegative matrix factorization methods for clustering. International conference on data mining
Lin Z, Jiang ZL, Davis LS (2009) Recognizing actions by shape-motion prototype trees. International conference on computer vision
Ling HB, Soatto S (2007) Proximity distribution kernels for geometric context in category recognition. International conference on computer vision
Liu HW, Feris R, Krueger V, Sun MT (2010) Unsupervised action classification using space-time link analysis. EURASIP J Image Video Process
Makris D, Ellis T (2002) Spatial and probabilistic modeling of pedestrian behavior. British machine vision conference
Mannila H, Toivonen H, Verkamo AI (1997) Discovery of frequent episodes in event sequences. Data Min Knowl Discov 1(3): 259–289
Morup M, Hansen LK, Arnfred SM (2006) Decomposing the time-frequency representation of EEG using nonnegative matrix and multi-way factorization. Technical report, Institute for Mathematical Modeling, Technical University of Denmark
Niebles JC, Wang HC, Fei-Fei L (2008) Unsupervised learning of human action categories using spatial-temporal words. Int J Comput Vis 79(3): 299–318
Robertson N, Reid I (2006) A general method for human activity recognition in video. J Comput Vis Image Underst 104(2): 232–248
Stauffer C, Grimson W (2000) Learning patterns of activity using real-time tracking. IEEE Trans Pattern Anal Mach Intell 22(8): 747–758
Sun XH, Chen MY, Hauptmann A (2009) Action recognition via local descriptors and holistic features. International conference on computer vision and pattern recognition
Tao DC, Li XL, Wu XD, Hu WM, Maybank SJ (2007a) Supervised tensor learning. Knowl Inf Syst 13: 1–42
Tao DC, Li XL, Wu XD, Maybank SJ (2007b) General tensor discriminant analysis and gabor features for gait recognition. IEEE Trans Pattern Anal Mach Intell 29: 1700–1715
Tao D, Li X, Wu X, Maybank S (2008a) Tensor rank one discriminant analysis—a convergent method for discriminative multilinear subspace selection. Neurocomputing 71: 1866–1882
Tao D, Song M, Li X, Shen J, Sun J, Wu X, Faloutsos C (2008b) Bayesian tensor approach for 3-D face modeling. IEEE Trans Circ Syst Video Tech 18(10): 1397–1410
Vaswani N, Chowdhury A, Chellappa R (2003) Activity recognition using the dynamics of the configuration of interacting objects. International conference on computer vision and pattern recognition
Wang F, Li T (2007) Gene Selection via Matrix Factorization. International symposium on bioinformatics and bioengineering
Wang F, Li T, Zhang CS (2008) Semi-supervised clustering via matrix factorization. SIAM conference on data mining
Xiang T, Gong S (2005) Video behaviour profiling and abnormality detection without manual labelling. International conference on computer vision
Xiang T, Gong S (2006) Beyond tracking: modelling action and understanding behavior. Int J Comput Vis 67(1): 21–51
Xu W, Liu X, Gong YH (2003) Document clustering based on non-negative matrix factorization. SIGIR conference on research and development in informaion retrieval
Yamamoto M, Mitomi H, Fujiwara F, Sato T (2006) Bayesian classification of task-oriented actions based on stochastic contextfree grammar. International conference on automatic face and gesture recognition
Yilmaz A, Shah M (2005) Actions sketch: a novel action representation. International conference on computer vision and pattern recognition
Zelnik-Manor L, Irani M (2001) Event-based video analysis. International conference on computer vision and pattern recognition
Zhong H, Shi J, Visontai M (2004) Detecting unusual activity in video. International conference on computer vision and pattern recognition
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Cui, P., Liu, ZQ., Sun, LF. et al. Hierarchical visual event pattern mining and its applications. Data Min Knowl Disc 22, 467–492 (2011). https://doi.org/10.1007/s10618-010-0195-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10618-010-0195-5