Skip to main content
Log in

Spatiotemporal Deformable Prototypes for Motion Anomaly Detection

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

This paper presents an approach for motion-based anomaly detection, where a prototype pattern is detected and elastically registered against a test sample to detect anomalies in the test sample. The prototype model is learned from multiple sequences to define accepted variations. “Supertrajectories” based on hierarchical clustering of dense point trajectories serve as an efficient and robust representation of motion patterns. An efficient hashing approach provides transformation hypotheses that are refined by a spatiotemporal elastic registration. We propose a new method for elastic registration of 3D+time trajectory patterns that induces spatial elasticity from trajectory affinities. The method is evaluated on a new motion anomaly dataset of juggling patterns and performs well in detecting subtle anomalies. Moreover, we demonstrate the applicability to biological motion patterns.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Notes

  1. Unusual crowd activity dataset made available by the University of Minnesota at: http://mha.cs.umn.edu/Movies/Crowd-Activity-All.avi.

  2. We thank the authors for providing essential code pieces to reimplement their method.

References

  • Adam, A., Rivlin, E., Shimshoni, I., & Reinitz, D. (2008). Robust real-time unusual event detection using multiple fixed-location monitors. IEEE Transactions Pattern Analysis and Machine Intelligence, 30(3), 555–560.

    Article  Google Scholar 

  • Antic, B. & Ommer, B. (2011). Video parsing for abnormality detection. In D. N. Metaxas, L. Quan, A. Sanfeliu, & L. J. V. Gool (Eds.), IEEE international conference on computer vision (ICCV) (pp. 2415–2422).

  • Benezeth, Y., Jodoin, P. M., Saligrama, V., & Rosenberger, C. (2009). Abnormal events detection based on spatio-temporal co-occurences. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 2458–2465).

  • Bensch, R., Brox, T., & Ronneberger, O. (2015). Spatiotemporal deformable prototypes for motion anomaly detection. In M. W. J. Xianghua Xie & G. K. L. Tam (Eds.), British machine vision conference (BMVC) (pp. 189.1–189.12). London: BMVA Press.

  • Besl, P., & McKay, N. D. (1992). A method for registration of 3-D shapes. IEEE Transactions Pattern Analysis and Machine Intelligence, 14(2), 239–256.

    Article  Google Scholar 

  • Boiman, O. & Irani, M. (2007b). Similarity by composition. In B. Schölkopf, J. Platt, & T. Hoffman (Eds.), Advances in neural information processing systems 19 (NIPS) (pp. 177–184). Cambridge, MA: MIT Press.

  • Boiman, O., & Irani, M. (2007a). Detecting irregularities in images and in video. International Journal of Computer Vision, 74(1), 17–31.

    Article  Google Scholar 

  • Brox, T., & Malik, J. (2011). Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Transactions Pattern Analysis and Machine Intelligence, 33(3), 500–513.

    Article  Google Scholar 

  • Byrd, R. H., Lu, P., Nocedal, J., & Zhu, C. (1995). A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5), 1190–1208.

    Article  MathSciNet  MATH  Google Scholar 

  • Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), 15:1–15:58.

    Article  Google Scholar 

  • Cong, Y., Yuan, J., & Liu, J. (2011). Sparse reconstruction cost for abnormal event detection. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 3449–3456).

  • Cong, Y., Yuan, J., & Tang, Y. (2013). Video anomaly search in crowded scenes via spatio-temporal motion context. IEEE Transactions on Information Forensics and Security, 8(10), 1590–1599.

    Article  Google Scholar 

  • Dee, H. M. & Hogg, D. C. (2004). Detecting inexplicable behaviour. In British machine vision conference (BMVC) (pp. 477–486).

  • Everitt, B., Landau, S., Leese, M., & Stahl, D. (2011). Cluster analysis. Wiley series in probability and statistics New York: Wiley.

  • Hu, W., Xiao, X., Fu, Z., Xie, D., Tan, T., & Maybank, S. (2006). A system for learning statistical motion patterns. IEEE Transactions Pattern Analysis and Machine Intelligence, 28(9), 1450–1464.

    Article  Google Scholar 

  • Kim, J. & Grauman, K. (2009). Observe locally, infer globally: A space-time MRF for detecting abnormal activities with incremental updates. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 2921–2928).

  • Kratz, L. & Nishino, K. (2009). Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 1446–1453).

  • Li, C., Han, Z., Ye, Q., & Jiao, J. (2013). Visual abnormal behavior detection based on trajectory sparse reconstruction analysis. Neurocomputing, 119,94–100. Intelligent Processing Techniques for Semantic-based Image and Video Retrieval.

  • Mahadevan, V., Li, W., Bhalodia, V., & Vasconcelos, N. (2010). Anomaly detection in crowded scenes. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 1975–1981).

  • Mehran, R., Oyama, A., & Shah, M. (2009). Abnormal crowd behavior detection using social force model. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 935–942).

  • Nait-Charif, H. & McKenna, S. J. (2004). Activity summarisation and fall detection in a supportive home environment. In IEEE international conference on pattern recognition (ICPR) (Vol. 4, pp. 323–326).

  • Piciarelli, C., Micheloni, C., & Foresti, G. L. (2008). Trajectory-based anomalous event detection. IEEE Transactions on Circuits and Systems for Video Technology, 18(11), 1544–1554.

    Article  Google Scholar 

  • Popoola, O. P., & Wang, K. (2012). Video-based abnormal human behavior recognition—A review. IEEE Transactions on Circuits and Systems for Video Technology, 42(6), 865–878.

    Google Scholar 

  • Roshtkhari, M. J., & Levine, M. D. (2013). An on-line, real-time learning method for detecting anomalies in videos using spatio-temporal compositions. Computer Vision and Image Understanding, 117(10), 1436–1452.

    Article  Google Scholar 

  • Saligrama, V. & Chen, Z. (2012). Video anomaly detection based on local statistical aggregates. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 2112–2119).

  • Saligrama, V., Konrad, J., & Jodoin, P. M. (2010). Video anomaly identification. IEEE Signal Processing Magazine, 27(5), 18–33.

    Article  Google Scholar 

  • Schmid, B., Shah, G., Scherf, N., Weber, M., Thierbach, K., Campos, C. P., Roeder, I., Aanstad, P., & Huisken, J. (2013). High-speed panoramic light-sheet microscopy reveals global endodermal cell dynamics. Nature Communications, 4. doi:10.1038/ncomms3207.

  • Sillito, R. R. & Fisher, R. B. (2008). Semi-supervised learning for anomalous trajectory detection. In British machine vision conference (BMVC) (pp. 1035–1044).

  • Sundaram, N., Brox, T., & Keutzer, K. (2010). Dense point trajectories by GPU-accelerated large displacement optical flow. In European conference on computer vision (ECCV). Lecture notes in computer science (pp. 438–451). New York: Springer.

  • Umeyama, S. (1991). Least-squares estimation of transformation parameters between two point patterns. IEEE Signal Processing Magazine, 13(4), 376–380.

    Google Scholar 

  • Winkelbach, S., Molkenstruck, S., & Wahl, F. M. (2006). Low-cost laser range scanner and fast surface registration approach. In Pattern Recognition (Proc. DAGM). Lecture notes in computer science (pp. 718–728). New York: Springer.

  • Wu, S., Moore, B. E., & Shah, M. (2010). Chaotic invariants of Lagrangian particle trajectories for anomaly detection in crowded scenes. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 2054–2060).

Download references

Acknowledgments

We thank J. Koch, A. Krämer, T. Paxian and D. Mai who contributed their juggling expertise and agreed to perform diverse juggling patterns in front of our Kinect camera. This study was supported by the Excellence Initiative of the German Federal and State Governments (BIOSS Centre for Biological Signalling Studies EXC 294 to R.B., T.B. and O.R.). N.S. and J.H. have received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (Grant agreement No. 647885).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Robert Bensch.

Additional information

Communicated by Xianghua Xie, Mark Jones, Gary Tam.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (m4v 22775 KB)

Appendix: Juggling Patterns Dataset Overview

Appendix: Juggling Patterns Dataset Overview

A tabular overview of the juggling pattern test set used in the experiments in Sect. 7.2 is given in Table 2. It lists all test sequences and their properties, such as the number of frames, distance to the juggler and viewing angle.

Table 2 Juggling patterns test set overview

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bensch, R., Scherf, N., Huisken, J. et al. Spatiotemporal Deformable Prototypes for Motion Anomaly Detection. Int J Comput Vis 122, 502–523 (2017). https://doi.org/10.1007/s11263-016-0934-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-016-0934-1

Keywords

Navigation