Skip to main content
SpringerLink
Log in

Spatiotemporal deep networks for detecting abnormality in videos

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Detecting and localizing anomalous behavior in the surveillance video is explored and spatiotemporal model, which jointly learns the appearance and motion-based feature is proposed. The general solution is to learn from the normal-data as reference models and uses various hands designed features. However, huge variations can occur within normal-behavior patterns. It is a challenge to represent higher-level concepts of a normal or abnormal event explicitly from raw input data. In the proposed framework, spatiotemporal features learned at various hidden layer are analyzed. Based on the learned representation, the reconstruction of video volumes are performed. Finally, the structural distortion based abnormality score is computed by considering luminance, contrast, and structural information to detect the presence of abnormality and localize them. Further, we also explored the performance of GMM and one-class SVM in a given scenario. The proposed structural distortion based abnormality detection and localization are evaluated on the publicly available UCSD and UMN dataset. The performance of the developed system is found to outperform the existing state-of-art methods for detecting and localizing abnormality at the frame as well as pixel-level. Recently, deep architecture is also found to be vulnerable to adversarial attacks and can easily be tricked to fool the system. However, most of the existing attacks are designed for the classification task. In this work, we utilize the gradient-based approach to generate adversarial samples for an abnormality detection system. Finally, we build the defense mechanism to detect the abnormality in the presence of such adversarial attacks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Notes

  1. UCSD dataset can be download from: http://www.svcl.ucsd.edu/projects/anomaly/dataset.htm

  2. UMN dataset can be download from http://mha.cs.umn.edu/proj_events.shtml

  3. Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks, Available at, http://www.cs.toronto.edu/~hinton/absps/science_som.pdf

  4. MeVisLab can be download from: http://www.mevislab.de/download/

  5. Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks, Available at, http://www.cs.toronto.edu/~hinton/absps/science_som.pdf

  6. https://github.com/cjlin1/libsvm

  7. Understanding Error Rates in Biometric Access Control, Available at http://www.ibfoundation.com/downloads/

  8. https://medium.com/onfido-tech/adversarial-attacks-and-defences-for-convolutional-neural-networks-66915ece52e7

References

  1. Aggarwal C C (2013) Outlier analysis. Springer Science & Business Media

  2. Akhtar N, Mian A (2018) Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access 6:14410–14430

    Article  Google Scholar 

  3. Andrews J T, Tanay T, Morton E J, Griffin L D (2016) Transfer representation-learning for anomaly detection. In: International conference on machine learning

  4. Basharat A, Gritai A, Shah M (2008) Learning object motion patterns for anomaly detection and improved object detection. In: Computer vision and pattern recognition, pp 1–8. IEEE

  5. Bengio Y (2009) Learning deep architectures for AI. Foundations and trends®in Machine Learning 2(1):1–127

    Article  Google Scholar 

  6. Bengio Y, Courville A, Vincent P (2013) Representation learning: A review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828

    Article  Google Scholar 

  7. Bengio Y, Goodfellow I, Courville A (2015) Deep learning. http://www.iro.umontreal.ca/bengioy/dlbook. Book in preparation for MIT Press

  8. Bengio Y, Lamblin P, Popovici D, Larochelle H, et al. (2007) Greedy layer-wise training of deep networks. Adv Neural Inform Process Syst 19:153

    Google Scholar 

  9. Biswas S, Babu R V (2015) Anomaly detection in compressed h.264/avc video. Multimed Tools Appl 74(24):11099–11115

    Article  Google Scholar 

  10. Carlini N, Wagner D (2017) Towards evaluating the robustness of neural networks. In: 2017 IEEE symposium on security and privacy (SP), pp 39–57. IEEE

  11. Chalapathy R, Chawla S (2019) Deep learning for anomaly detection: A survey. arXiv:1901.03407

  12. Chalapathy R, Menon A K, Chawla S (2018) Anomaly detection using one-class neural networks. arXiv:1802.06360

  13. Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: A survey. ACM Comput Surv (CSUR) 41(3):15

    Article  Google Scholar 

  14. Choi M J, Torralba A, Willsky A S (2012) Context models and out-of-context objects. Pattern Recogn Lett 33(7):853–862

    Article  Google Scholar 

  15. Chong Y S, Tay Y H (2015) Modeling representation of videos for anomaly detection using deep learning: A review. arXiv:1505.00523

  16. Cong Y, Yuan J, Liu J (2011) Sparse reconstruction cost for abnormal event detection. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp 3449–3456. IEEE

  17. Deng L (2012) Three classes of deep learning architectures and their applications: A tutorial survey. APSIPA transactions on signal and information processing

  18. Fawzi A, Fawzi H, Fawzi O (2018) Adversarial vulnerability for any classifier. In: Advances in neural information processing systems, pp 1178–1187

  19. Fischer A, Igel C (2014) Training restricted boltzmann machines: An introduction. Pattern Recogn 47(1):25–39

    Article  Google Scholar 

  20. Gondim-Ribeiro G, Tabacof P, Valle E (2018) Adversarial attacks on variational autoencoders. arXiv:1806.04646

  21. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  22. Goodfellow I J, Shlens J, Szegedy C (2014) Explaining and harnessing adversarial examples. arXiv:1412.6572

  23. Hawkins S, He H, Williams G, Baxter R (2002) Outlier detection using replicator neural networks. In: Data warehousing and knowledge discovery, pp 170–180. Springer

  24. Hinton G (2010) A practical guide to training restricted boltzmann machines. Momentum 9(1):926

    Google Scholar 

  25. Hinton G, Osindero S, Teh Y W (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MathSciNet  Google Scholar 

  26. Hinton G E, Salakhutdinov R R (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  Google Scholar 

  27. Hu W, Tan T, Wang L, Maybank S (2004) A survey on visual surveillance of object motion and behaviors. IEEE Trans Syst Man Cybern Part C Appl Rev 34 (3):334–352

    Article  Google Scholar 

  28. Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259

    Article  Google Scholar 

  29. Javan-Roshtkhari M (2014) Visual event description in videos. McGill University, Ph.D. thesis

    Google Scholar 

  30. Kiran B, Thomas D, Parakkal R (2018) An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. J Imaging 4(2):36

    Article  Google Scholar 

  31. Kos J, Fischer I, Song D (2018) Adversarial examples for generative models. In: 2018 IEEE security and privacy workshops (SPW), pp 36–42. IEEE

  32. Krizhevsky A, Sutskever I, Hinton G E (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  33. Kumaran S K, Dogra D P, Roy P P (2019) Anomaly detection in road traffic using visual surveillance: A survey. arXiv:1901.08292

  34. Kurakin A, Goodfellow I, Bengio S (2016) Adversarial examples in the physical world. arXiv:1607.02533

  35. Leach M J, Sparks E P, Robertson N M (2014) Contextual anomaly detection in crowded surveillance scenes. Pattern Recogn Lett 44:71–79

    Article  Google Scholar 

  36. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521 (7553):436–444

    Article  Google Scholar 

  37. Li N, Wu X, Xu D, Guo H, Feng W (2015) Spatio-temporal context analysis within video volumes for anomalous-event detection and localization. Neurocomputing 155:309–319

    Article  Google Scholar 

  38. Li W, Mahadevan V, Vasconcelos N (2014) Anomaly detection and localization in crowded scenes. IEEE Trans Pattern Anal Mach Intell 36(1):18–32

    Article  Google Scholar 

  39. Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. In: IEEE conference on computer vision and pattern recognition (CVPR), 2010, pp 1975–1981. https://doi.org/10.1109/CVPR.2010.5539872

  40. Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. In: IEEE conference on computer vision and pattern recognition, 2009. CVPR 2009, pp 935–942. IEEE

  41. Mo D (2012) A survey on deep learning: One small step toward AI. Dept. Computer Science, Univ. of New Mexico USA

  42. Papernot N, McDaniel P, Goodfellow I, Jha S, Celik Z B, Swami A (2017) Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp 506–519. ACM

  43. Popoola O P, Wang K (2012) Video-based abnormal human behavior recognition-a review. IEEE Trans Syst Man Cybern Part C: Appl Rev 42(6):865–878

    Article  Google Scholar 

  44. Ravanbakhsh M, Nabi M, Mousavi H, Sangineto E, Sebe N (2016) Plug-and-play cnn for crowd motion analysis: An application in abnormal event detection. arXiv:1610.00307

  45. Reddy V, Sanderson C, Lovell B C (2011) Improved anomaly detection in crowded scenes via cell-based analysis of foreground speed, size and texture. In: Computer vision and pattern recognition workshops, pp 55–61

  46. Richardson I E (2004) H.264 and MPEG-4 video compression: Video coding for next-generation multimedia. John Wiley & Sons

  47. Roshtkhari M J, Levine M D (2013) An on-line, real-time learning method for detecting anomalies in videos using spatio-temporal compositions. Comput Vis Image Underst 117(10):1436–1452

    Article  Google Scholar 

  48. Roshtkhari M J, Levine M D (2013) Online dominant and anomalous behavior detection in videos. In: Proceedings of computer vision and pattern recognition (CVPR), pp 2611–2618. IEEE

  49. Ryan D, Denman S, Fookes C, Sridharan S (2011) Textures of optical flow for real-time anomaly detection in crowds. In: 2011 8th IEEE international conference on advanced video and signal-based surveillance (AVSS), pp 230–235. IEEE

  50. Sabokrou M, Fayyaz M, Fathy M, Moayed Z, Klette R (2018) Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes. Comput Vis Image Underst 172:88–97

    Article  Google Scholar 

  51. Saleh B, Farhadi A, Elgammal A (2013) Object-centric anomaly detection by attribute-based reasoning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 787–794

  52. Sharma MK, Sheet D, Biswas PK (2019) Image embedding for detecting irregularity. In: Proceedings of 3rd international conference on computer vision and image processing: CVIP 2018, vol 2. Springer Nature, Berlin, pp 243

  53. Sharma M K, Sheet D, Biswas P K (2016) Abnormality detecting deep belief network. In: Proceedings of the international conference on advances in information communication technology & computing, pp 11. ACM

  54. Sodemann A A, Ross M P, Borghetti B J (2012) A review of anomaly detection in automated surveillance. IEEE Trans Syst Man Cybern Part C: Appl Rev 42(6):1257–1272

    Article  Google Scholar 

  55. Tan H, Zhai Y, Liu Y, Zhang M (2016) Fast anomaly detection in traffic surveillance video based on robust sparse optical flow. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 1976–1980. IEEE

  56. Wang C, Yao H, Sun X (2017) Anomaly detection based on spatio-temporal sparse representation and visual attention analysis. Multimed Tools Appl 76(5):6263–6279

    Article  Google Scholar 

  57. Wang Z, Bovik A C (2009) Mean squared error: Love it or leave it? a new look at signal fidelity measures. IEEE Signal Process Mag 26(1):98–117

    Article  Google Scholar 

  58. Wang Z, Bovik A C, Sheikh H R, Simoncelli E P (2004) Image quality assessment: From error visibility to structural similarity. IEEE Trans Image Process. 13(4):600–612

    Article  Google Scholar 

  59. Williams G, Baxter R, He H, Hawkins S, Gu L (2002) A comparative study of RNN for outlier detection in data mining. In: Proceedings of 13th international conference on data mining, pp 709–709. IEEE Computer Society

  60. Xu D, Ricci E, Yan Y, Song J, Sebe N (2015) Learning deep representations of appearance and motion for anomalous event detection. arXiv:1510.01553

  61. Yan W, Yu L (2019) On accurate and reliable anomaly detection for gas turbine combustors: A deep learning approach. arXiv:1908.09238

  62. Yong S P, Deng J D, Purvis M K (2012) Novelty detection in wildlife scenes through semantic context modelling. Pattern Recogn 45(9):3439–3450

    Article  Google Scholar 

  63. Yuan X, He P, Zhu Q, Li X (2019) Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems

  64. Zahangir Alom M, Taha T M, Yakopcic C, Westberg S, Sidike P, Shamima Nasrin M, Van Esesn B C, Awwal A A S, Asari V K (2018) The history began from alexnet: A comprehensive survey on deep learning approaches. arXiv:1803.01164

  65. Zaharescu A, Wildes R (2010) Anomalous behaviour detection using spatiotemporal oriented energies, subset inclusion histogram comparison and event-driven processing. In: Computer Vision–ECCV 2010, pp 563–576. Springer

  66. Zhang T, Lu H, Li S Z (2009) Learning semantic scene models by object classification and trajectory clustering. In: Computer vision and pattern recognition, pp 1940–1947. IEEE

Download references

Author information

Authors and Affiliations

  1. IIT Kharagpur, Kharagpur, India

    M. K. Sharma, D. Sheet & P. K. Biswas

Authors
  1. M. K. Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. D. Sheet
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. P. K. Biswas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. K. Sharma.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, M.K., Sheet, D. & Biswas, P.K. Spatiotemporal deep networks for detecting abnormality in videos. Multimed Tools Appl 79, 11237–11268 (2020). https://doi.org/10.1007/s11042-020-08786-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08786-w

Keywords

Search

Navigation