Skip to main content

Advertisement

Springer Nature Link
Log in

DeMAAE: deep multiplicative attention-based autoencoder for identification of peculiarities in video sequences

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In videos, anomaly detection is challenging due to its diverse nature in different application domains. Reconstruction and prediction-based methods have been widely employed to detect anomalies. Due to the generalization capability of a deep neural network, sometimes, it recreates irregular patterns along with regular ones. This paper presents a novel autoencoder-based framework called deep multiplicative attention-based autoencoder (DeMAAE) to detect anomalies in a video sequence. The global attention mechanism is used at the decoder side of DeMAAE for better feature learning during the decoding phase. An attention map is created by taking the dot product between all encoder’s hidden states and the previously generated decoder’s hidden state. After that, the final output of the decoder is determined by the context vector. The context vector is computed using the weighted summation of all encoder’s hidden states and attention weight. DeMAAE delivers an improved runtime of 0.015 s (\( \sim \) 67 fps) for detecting anomalies during testing. Extensive experiments have been performed on the two diversified and widely used datasets (UCSD Pedestrian and CUHK Avenue) to compare the efficacy of DeMAAE with different state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Liu, W., Luo, W., Lian, D., Gao, S.: Future frame prediction for anomaly detection—a new baseline. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6536–6545 (2018)

  2. Luo, W.: A revisit of sparse coding based anomaly detection in stacked RNN framework. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 341–349 (2017)

  3. Dalal, N., Triggs, B., Schmid, C.: Human detection using oriented histograms of flow and appearance. In: European Conference on Computer Vision, pp. 428–441. Springer (2006)

  4. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pp. 886–893. IEEE (2005)

  5. Wu, S., Moore, B.E., Shah, M.: Chaotic invariants of lagrangian particle trajectories for anomaly detection in crowded scenes. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2054–2060. IEEE (2010)

  6. Jiang, F., Yuan, J., Tsaftaris, S.A., Katsaggelos, A.K.: Anomalous video event detection using spatiotemporal context. Comput. Vis. Image Underst. 115(3), 323–333 (2011)

  7. Kim, J., Grauman, K.: Observe locally, infer globally: a space-time MRF for detecting abnormal activities with incremental updates. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2928. IEEE (2009)

  8. Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A.K., Davis, L.S.: Learning temporal regularity in video sequences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 733–742 (2016)

  9. Aslam, N., Kolekar, M.H.: Unsupervised anomalous event detection in videos using spatio-temporal inter-fused autoencoder. Multimedia Tools and Applications, pp. 1–26 (2022)

  10. Chong, Y.S., Tay, Y.H.: Abnormal event detection in videos using spatiotemporal autoencoder. In: International Symposium on Neural Networks, pp. 189–196. Springer (2017)

  11. Aslam, N., Rai, P.K., Kolekar, M.H.: A3n: Attention-based adversarial autoencoder network for detecting anomalies in video sequence. J. Vis. Commun. Image Represent., 87:103598 (2022)

  12. Medel, J.R., Savakis, A..: Anomaly detection in video using predictive convolutional long short-term memory networks. arXiv preprint arXiv:1612.00390 (2016)

  13. Liu, Z., Nie, Y., Long, C., Zhang, Q., Li, G..: A hybrid video anomaly detection framework via memory-augmented flow reconstruction and flow-guided frame prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13588–13597 (2021)

  14. Astrid, M., Zaheer, M.Z., Lee, S.-I.: Synthetic temporal anomaly guided end-to-end video anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 207–214 (2021)

  15. Pang, G., Yan, C., Shen, C., van den H., Anton, B.X.: Self-trained deep ordinal regression for end-to-end video anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12173–12182 (2020)

  16. Feng, J.-C., Hong, F.-T., Zheng, W.-S..: Mist: multiple instance self-training framework for video anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14009–14018 (2021)

  17. Song, H., Sun, C., Wu, X., Chen, M., Jia, Y.: Learning normal patterns via adversarial attention-based autoencoder for abnormal event detection in videos. IEEE Trans. Multimedia 22(8), 2138–2148 (2019)

  18. Kumar, D., Bezdek, J.C., Rajasegarar, S., Leckie, C., Palaniswami, M.: A visual-numeric approach to clustering and anomaly detection for trajectory data. Vis. Comput. 33(3), 265–281 (2017)

    Article  Google Scholar 

  19. Li, Q., Wang, Y., Sharf, A., Cao, Y., Tu, C., Chen, B., Yu, S.: Classification of gait anomalies from kinect. Vis. Comput. 34(2), 229–241 (2018)

  20. Bansod, S.D., Nandedkar, A.V.: Crowd anomaly detection and localization using histogram of magnitude and momentum. Vis. Comput. 36(3), 609–620 (2020)

    Article  Google Scholar 

  21. Coşar, S., Donatiello, G., Bogorny, V., Garate, C., Alvares, L.O., Brémond, F.: Toward abnormal trajectory and event detection in video surveillance. IEEE Trans. Circuits Syst. Video Technol. 27(3), 683–695 (2016)

  22. Zhang, J., Wang, Z., Meng, J., Tan, Y.-P., Yuan, J.: Boosting positive and unlabeled learning for anomaly detection with multi-features. IEEE Trans. Multimedia 21(5), 1332–1344 (2018)

  23. Mahadevan, V., Li, W., Bhalodia, V., Vasconcelos, N.: Anomaly detection in crowded scenes. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1975–1981. IEEE (2010)

  24. Aslam N., Sharma, V.: Foreground detection of moving object using Gaussian mixture model. In: 2017 International Conference on Communication and Signal Processing (ICCSP), pp. 1071–1074. IEEE (2017)

  25. Lu, C., Shi, J., Jia, J.: Abnormal event detection at 150 fps in matlab. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2720–2727 (2013)

  26. Zhao, B., Fei-Fei, L., Xing, E.P.: Online detection of unusual events in videos via dynamic sparse coding. In: CVPR 2011, pp. 3313–3320. IEEE (2011)

  27. Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6479–6488 (2018)

  28. Georgescu, M.-I., Barbalau, A., Ionescu, R.T., Khan, F.S., Popescu, M., Shah, M.: Anomaly detection in video via self-supervised and multi-task learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12742–12752 (2021)

  29. Sabokrou, M., Fayyaz, M., Fathy, M., Klette, R.: Deep-cascade: cascading 3d deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans. Image Process. 26(4), 1992–2004 (2017)

    Article  MathSciNet  PubMed  ADS  Google Scholar 

  30. Zhao, Y., Zhou, L., Fu, K., Yang, J.: Abnormal event detection using spatio-temporal feature and nonnegative locality-constrained linear coding. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 3354–3358. IEEE (2016)

  31. Cheng, D., Zhou, J., Wang, N., Gao, X.: Hybrid dynamic contrast and probability distillation for unsupervised person re-id. IEEE Trans. Image Process. 31, 3334–3346 (2022)

    Article  PubMed  ADS  Google Scholar 

  32. Tudor Ionescu, R., Smeureanu, S., Alexe, B., Popescu, M.: Unmasking the abnormal events in video. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2895–2903 (2017)

  33. Xi, P., Guan, H., Shu, C., Borgeat, L., Goubran, R.: An integrated approach for medical abnormality detection using deep patch convolutional neural networks. Vis. Comput. 36(9), 1869–1882 (2020)

    Article  Google Scholar 

  34. Luong, M.-T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015)

  35. Ba, J., Mnih, V., Kavukcuoglu, K.: Multiple object recognition with visual attention. arXiv preprint arXiv:1412.7755 (2014)

  36. Zhang, Y., Li, J., Wu, G., Zhang, H., Shi, Z., Liu, Z., Wu, Z., Jiang, N.: Temporal transformer networks with self-supervision for action recognition. arXiv preprint arXiv:2112.07338 (2021)

  37. Zhou, J.T., Zhang, L., Fang, Z., Du, J., Peng, X., Xiao, Y.: Attention-driven loss for anomaly detection in video surveillance. IEEE Trans. Circuits Syst. Video Technol. 30(12), 4639–4647 (2019)

    Article  Google Scholar 

  38. Zheng, L., Li, Z., Li, J., Li, Z., Gao, J.: Addgraph: anomaly detection in dynamic graph using attention-based temporal GCN. In: IJCAI, pp. 4419–4425 (2019)

  39. Ma, H., Zhang, L.: Attention-based framework for weakly supervised video anomaly detection. J. Supercomput. 1–21 (2022)

  40. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  41. Ravanbakhsh, M., Nabi, M., Sangineto, E., Marcenaro, L., Regazzoni, C., Sebe, N..: Abnormal event detection in videos using generative adversarial nets. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 1577–1581. IEEE (2017)

  42. Yang, B., Cao, J., Wang, N., Liu, X.: Anomalous behaviors detection in moving crowds based on a weighted convolutional autoencoder-long short-term memory network. IEEE Trans. Cogn. Dev. Syst. 11(4), 473–482 (2018)

    Article  Google Scholar 

  43. Gong, D., Liu, L., Le, V., Saha, B., Mansour, M.R., Venkatesh, S., van den Hengel, A.: Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1705–1714 (2019)

  44. Ramachandra, B., Jones, M.: Street scene: a new dataset and evaluation protocol for video anomaly detection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2569–2578 (2020)

  45. Yan, S., Smith, J.S., Lu, W., Zhang, B.: Abnormal event detection from videos using a two-stream recurrent variational autoencoder. IEEE Trans. Cogn. Dev. Syst. 12(1), 30–42 (2018)

    Article  Google Scholar 

  46. Nawaratne, R., Alahakoon, D., De Silva, D., Yu, X.: Spatiotemporal anomaly detection using deep learning for real-time video surveillance. IEEE Trans. Ind. Inf. 16(1), 393–402 (2019)

    Article  Google Scholar 

  47. Wang, X., Xie, W., Song, J.: Learning spatiotemporal features with 3D CNN and convgru for video anomaly detection. In: 2018 14th IEEE International Conference on Signal Processing (ICSP), pp. 474–479. IEEE (2018)

  48. Zhao, Y., Deng, B., Shen, C., Liu, Y., Lu, H., Hua, X.-S.: Spatio-temporal autoencoder for video anomaly detection. In: Proceedings of the 25th ACM International Conference on Multimedia, pp. 1933–1941 (2017)

  49. Luo, W., Liu, W., Gao, S.: Remembering history with convolutional LSTM for anomaly detection. In: 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 439–444. IEEE (2017)

  50. Park, H., Noh, J., Ham, B.: Learning memory-guided normality for anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14372–14381 (2020)

  51. Ji, X., Li, B., Zhu, Y.: Tam-net: temporal enhanced appearance-to-motion generative network for video anomaly detection. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2020)

  52. Wang, L., Zhou, F., Li, Z., Zuo, W., Tan, H.: Abnormal event detection in videos using hybrid spatio-temporal autoencoder. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 2276–2280. IEEE (2018)

  53. Zhang, Y., Nie, X., He, R., Chen, M., Yin, Y.: Normality learning in multispace for video anomaly detection. IEEE Trans. Circuits Syst. Video Technol. (2020)

  54. Ye, M., Peng, X., Gan, W., Wu, W., Qiao, Y.: Anopcn: video anomaly detection via deep predictive coding network. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 1805–1813 (2019)

Download references

Author information

Authors and Affiliations

  1. Video Surveillance Lab, Department of Electrical Engineering, Indian Institute of Technology Patna, Bihta, 801106, India

    Nazia Aslam

  2. Department of Electrical Engineering, Indian Institute of Technology Patna, Bihta, 801106, India

    Maheshkumar H. Kolekar

Authors
  1. Nazia Aslam
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Maheshkumar H. Kolekar
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Nazia Aslam.

Ethics declarations

Conflicts of interest

All authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aslam, N., Kolekar, M.H. DeMAAE: deep multiplicative attention-based autoencoder for identification of peculiarities in video sequences. Vis Comput 40, 1729–1743 (2024). https://doi.org/10.1007/s00371-023-02882-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02882-2

Keywords