Skip to main content
SpringerLink
Log in

Fast and accurate detection and localization of abnormal behavior in crowded scenes

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

This paper presents a novel video processing method for accurate and fast anomaly detection and localization in crowded scenes. We propose a cubic-patch-based method based on a cascade of classifiers, which combines the results of two types of video descriptors. Based on the low likelihood of an anomaly occurrence and the redundancy of structures in normal patches of a video, we introduce two efficient feature sets for describing the spatial and temporal video context, which are named “local” and “global” descriptors. The local description is based on the relation of a patch with its neighbors, and the global description is provided by a sparse auto-encoder. Two reference models, using the local and global descriptions of the training normal patches, are learned as two one-class classifiers. For being fast and accurate, these two classifiers are combined as a cascaded classifier. First, the local classifier, which is faster than the global one, is used for early identification of “many” normal cubic patches. Then, the remaining patches are checked “carefully” by the global classifier. Also, we propose a technique for learning from small patches and inferring from larger patches; this leads to an improved performance. It is shown that the proposed method performs comparable to, or even better than top-performing detection and localization methods on standard benchmarks but with a substantial improvement in speed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Cong, Y., et al.: Sparse reconstruction cost for abnormal event detection. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 3449–3456 (2011)

  2. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc. (2012)

  3. Girshick, R., et al.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 580–587

  4. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 568–576. Curran Associates, Inc. (2014)

  5. Yang, Y., et al.: Semi-supervised learning of feature hierarchies for object detection in a video. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 1650–1657 (2013)

  6. Giusti, A., et al.: Fast image scanning with deep max-pooling convolutional neural networks. In: IEEE International Conference on Image Processing, pp. 4034–4038 (2013)

  7. Bertini, M., Del Bimbo, A., Seidenari, L.: Multi-scale and real-time non-parametric approach for anomaly detection and localization. Comput. Vis. Image Underst. 116(3), 320–329 (2012)

    Article  Google Scholar 

  8. Sabokrou, M., et al.: Real-time anomaly detection and localization in crowded scenes. In: IEEE Conference on Computer Vision Pattern Recognition Workshops, pp. 320–329 (2015)

  9. Coates, A., et al.: An analysis of single-layer networks in unsupervised feature learning. In: Conference on Artificial Intelligence Statistics, pp. 215–223 (2011)

  10. Statistical Visual Computing Lab, UC San Diego: UCSD Anomaly Detection Dataset. www.svcl.ucsd.edu/projects/anomaly/dataset.html. Accessed Feb 2013 (2013)

  11. Artificial Intelligence, Robotics and Vision Laboratory University of Minnesota, Dep. Computer Science and Engineering: Monitoring Human Activity. http://mha.cs.umn.edu/Movies/Crowd-Activity-All.avimha.cs.umn.edu/Movies/Crowd-Activity-All.avi. Accessed Feb 2013 (2006)

  12. Jianga, F., Yuan, J., Tsaftarisa, S.A., Katsaggelosa, A.K.: Anomalous video event detection using spatiotemporal context. Comput. Vis. Image Underst. 115(3), 323–333 (2011)

    Article  Google Scholar 

  13. Wu, S., et al.: Chaotic invariants of Lagrangian particle trajectories for anomaly detection in crowded scenes. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 2054–2060 (2010)

  14. Piciarelli, C., Micheloni, C., Foresti, G.L.: Trajectory-based anomalous event detection. IEEE Trans. Circuits Syst. Video Technol. 18(11), 1544–1554 (2008)

    Article  Google Scholar 

  15. Antonakaki, P., Kosmopoulos, D., Perantonis, S.J.: Detecting abnormal human behaviour using multiple cameras. Signal Process. 89(9), 1723–1738 (2009)

    Article  MATH  Google Scholar 

  16. Piciarelli, C., Foresti, G.L.: On-line trajectory clustering for anomalous events detection. Pattern Recognit. Lett. 27(15), 1835–1842 (2006)

    Article  Google Scholar 

  17. Hu, W., Xiao, X., Fu, Z., et al.: A system for learning statistical motion patterns. IEEE Trans. Pattern Anal. Mach. Intell. 28(9), 1450–1464 (2006)

    Article  Google Scholar 

  18. Calderara, S., Heinemann, U., Prati, A., et al.: Detecting anomalies in people’s trajectories using spectral graph analysis. Comput. Vis. Image Underst. 115(8), 1099–1111 (2011)

    Article  Google Scholar 

  19. Morris, B.T., Trivedi, M.M.: Trajectory learning for activity understanding: unsupervised, multilevel, and long-term adaptive approach. IEEE Trans. Pattern Anal. Mach. Intell. 33(11), 2287–2301 (2011)

    Article  Google Scholar 

  20. Tung, F., Zelek, J.S., Clausi, D.A.: Goal-based trajectory analysis for unusual behaviour detection in intelligent surveillance. Image Vis. Comput. 29(4), 230–240 (2011)

    Article  Google Scholar 

  21. Biswas, S., Rabu, v: Anomaly detection via short local trajectories. Neurocomputing 242, 63–72 (2017)

    Article  Google Scholar 

  22. Adam, A., Rivlin, E., Shimshoni, I., Reinitz, D.: Robust real-time unusual event detection using multiple fixed location monitors. IEEE Trans. Pattern Anal. Mach. Intell. 30(3), 555–560 (2008)

    Article  Google Scholar 

  23. Boiman, O., Irani, M.: Detecting irregularities in images and in video. Int. J. Comput. Vis. 74(1), 17–31 (2007)

    Article  Google Scholar 

  24. Mahadevan, V., et al.: Anomaly detection in crowded scenes. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 1975–1981 (2010)

  25. Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 18–32 (2014)

    Article  Google Scholar 

  26. Kim, J., Grauman, K.: Observe locally, infer globally: a space–time MRF for detecting abnormal activities with incremental updates. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 2921–2928 (2009)

  27. Benezeth, Y., et al.: Abnormal events detection based on spatio-temporal co-occurrences. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 1446–1453 (2009)

  28. Kratz, L., Nishino, K.: Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 1446–1453 (2009)

  29. Mehran, R., et al.: Abnormal crowd behavior detection using social force model. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 935–942 (2009)

  30. Zaharescu, A., Wildes, R.: Anomalous behaviour detection using spatiotemporal oriented energies, subset inclusion histogram comparison and event-driven processing. European Conference on Computer Vision, LNCS, pp. 563–576 (2010)

  31. Saligrama, V., Chen, Z.: Video anomaly detection based on local statistical aggregates. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 2112–2119 (2012)

  32. Antić, B., Ommer, B.: Video parsing for abnormality detection. In: International Conference on Computer Vision, pp. 2415–2422 (2011)

  33. Reddy, V., et al.: Improved anomaly detection in crowded scenes via cell-based analysis of foreground speed, size and texture. In: CVPR Workshops, pp. 55–61 (2011)

  34. Ullah, H., Conci, N.: Crowd motion segmentation and anomaly detection via multi-label optimization. In: ICPR Workshop Pattern Recognition Crowd Analysis (2012)

  35. Roshtkhari, M.J., Levine, M.D.: An on-line, real-time learning method for detecting anomalies in videos using spatio-temporal compositions. Comput. Vis. Image Underst. 117(10), 1436–1452 (2013)

    Article  Google Scholar 

  36. Cong, Y., Yuan, J., Tang, Y.: Video anomaly search in crowded scenes via spatio-temporal motion context. IEEE Trans. Inf. Forensics Secur. 8(10), 1590–1599 (2013)

    Article  Google Scholar 

  37. Zhu, Y., et al.: Context-aware modeling and recognition of activities in video. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 2491–2498 (2013)

  38. Roshtkhari, M.J., Levine, M.D.: Online dominant and anomalous behavior detection in videos. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 2611–2618 (2013)

  39. Nakahata, M.T., Thomaz, L.A., da Silva, A.F., da Silva, E.A.B., Netto, S.L.: Anomaly detection with a moving camera using spatio-temporal codebooks. Multidim. Syst. Signal Process. (2017). doi:10.1007/s11045-017-0486-8

  40. Xu, D., Song, R., Wu, X., et al.: Video anomaly detection based on a hierarchical activity discovery within spatio-temporal contexts. Neurocomputing 143, 144–152 (2014)

    Article  Google Scholar 

  41. Cheng, K.W., et al.: Video anomaly detection and localization using hierarchical feature representation and Gaussian process regression. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 2909–2917 (2015)

  42. Ullah, H., Tenuti, L., Conci, N.: Gaussian mixtures for anomaly detection in crowded scenes. In: IS&T/SPIE Electronic Imaging, vol. 8663 (2013)

  43. Ullah, H., Tenuti, L., Conci, N.: Real-time anomaly detection in dense crowded scenes. In: IS&T/SPIE Electronic Imaging, vol. 9026 (2014)

  44. Ullah, H., Tenuti, L., Conci, N.: Dominant motion analysis in regular and irregular crowd scenes. In: ECCV Workshop Human Behaviour Understanding, pp. 62–72 (2014)

  45. Mousavi, H., et al.: Analyzing tracklets for the detection of abnormal crowd behavior. In: IEEE Winter Conference on Applications Computer Vision, pp. 148–155 (2015)

  46. Mousavi, H., et al.: Abnormality detection with improved histogram of oriented tracklets. In: International Conference on Image Analysis and Processing, vol. 9280, pp. 722–732 (2015)

  47. Rabiee, H., et al.: Detection and localization of crowd behavior using a novel tracklet-based mode. Int. J. Mach. Learn. Cybern. 8(36), 1–12 (2017)

    Google Scholar 

  48. Yuan, Y., Fang, J., Wang, Q.: Online anomaly detection in crowd scenes via structure analysis. IEEE Trans. Cybern. 45(3), 562–575 (2015)

    Article  Google Scholar 

  49. Lee, D., Suk, H.I., Park, S.K., et al.: Motion influence map for unusual human activity detection and localization in crowded scenes. IEEE Trans. Circuits Syst. Video Technol. 25(10), 1612–1623 (2015)

    Article  Google Scholar 

  50. Zhang, D., et al.: Semi-supervised adapted HMMS for unusual event detection. IEEE Conf. Computer Vis. Pattern Recognit. 1, 611–618 (2005)

    Google Scholar 

  51. Lu, C., et al.: Abnormal event detection at 150 fps in MATLAB. In: IEEE Internatioanal Conference on Computer Vision, pp. 2720–2727 (2013)

  52. Xiao, T., Zhang, C., Zha, H.: Learning to detect anomalies in surveillance video. IEEE Signal Process. Lett. 22(9), 1477–1481 (2015)

    Article  Google Scholar 

  53. Li, N., Wu, X., Xu, D., et al.: Spatio-temporal context analysis within video volumes for anomalous-event detection and localization. Neurocomputing 155, 309–319 (2015)

    Article  Google Scholar 

  54. Xu, D., et al.: Learning deep representations of appearance and motion for anomalous event detection. In: British Machine Vision Conference, pp. 8.1–8.12 (2015)

  55. Sabokrou, M., et al.: Video anomaly detection and localisation based on the sparsity and reconstruction error of auto-encoder. Electron. Lett. 52, 1122–1124 (2016)

    Article  Google Scholar 

  56. Sabokrou, M., et al.: Deep-cascade: cascading 3D deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans. Image Process. 26(4), 1992–2004 (2017)

    Article  MathSciNet  Google Scholar 

  57. Vincent, P. et al.: Extracting and composing robust features with denoising autoencoders. In: ACM International Conference on Machine Learning, pp. 1096–1103 (2008)

  58. Sabokrou, M., et al.: Deep-anomaly: fully convolutional neural network for fast anomaly detection in crowded scenes. Arxiv e-prints arXiv:1609.00866 (2017)

  59. Ravanbakhsh, M., et al.: Plug-and-play CNN for crowd motion analysis: an application in abnormal event detection. arXiv preprint arXiv:1610.00307 (2016)

  60. Zhou, S., et al.: Spatio-temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Process. Image Commun. 47, 358–368 (2016)

    Article  Google Scholar 

  61. Hu, X., et al.: Video anomaly detection using deep incremental slow feature analysis network. IET Comput. Vis. 10(4), 258–265 (2016)

    Article  Google Scholar 

  62. Fang, Z., et al.: Abnormal event detection in crowded scenes based on deep learning. Multimed. Tools Appl. 75, 14617–14639 (2016)

    Article  Google Scholar 

  63. Munwar, A., Vinayavekhin, P., Magistris, G.: Spatio-temporal anomaly detection for industrial robots through prediction in unsupervised feature space. In: IEEE Winter Conference on Applications Computer Vision, pp. 1017–1025 (2017)

  64. Brunet, D., Vrscay, E.R., Wang, Z.: On the mathematical properties of the structural similarity index. IEEE Trans. Image Process. 21(4), 1488–1499 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  65. Wang, Z., Bovik, A.C., Sheikh, H.R., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

Download references

Acknowledgements

This research was in part supported by a grant from IPM. (No. CS1396-5-01)

Author information

Authors and Affiliations

  1. School of Computer Science, Institute for Research in Fundamental Sciences (IPM), P.O. Box 19395-5746, Tehran, Iran

    Mohammad Sabokrou

  2. Iran University of Science and Technology, Narmak, Tehran, Iran

    Mahmood Fathy

  3. Auckland University of Technology, Auckland, New Zealand

    Zahra Moayed & Reinhard Klette

Authors
  1. Mohammad Sabokrou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mahmood Fathy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zahra Moayed
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Reinhard Klette
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammad Sabokrou.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sabokrou, M., Fathy, M., Moayed, Z. et al. Fast and accurate detection and localization of abnormal behavior in crowded scenes. Machine Vision and Applications 28, 965–985 (2017). https://doi.org/10.1007/s00138-017-0869-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-017-0869-8

Keywords

Search

Navigation