Skip to main content
SpringerLink
Log in

Multi-memory video anomaly detection based on scene object distribution

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

With the popularity of surveillance equipment and the rise of intelligent surveillance, video anomaly detection has gradually become a research hotspot. Among them, for video processing, the three-channel video frame data can be directly used as the input of model, or some motion information can be extracted from the video frame, such as calculating optical flow, and then motion information and video frame can be input into the model together for anomaly detection. However, since the amount of background information in the overall situation is far greater than that of object information, abnormal objects are not concerned. In addition, there ia a phenomenon that objects close to the camera are more likely to be judged as anomalous due to the difference in viewpoint resulting in different sizes of objects captured in the scene. This paper proposes a multi-memory video anomaly detection algorithm based on scene object distribution. Firstly, add local anomaly branch to the model, and use memory modules to explicitly model the multiple normal modes of the global frame and the local object; secondly, scale the object to the same measurement standard according to the scene object distribution, which alleviates the impact of the view difference; finally, considering the difficulty of anomaly positioning, a new anomaly location method that combines global anomalies and local anomalies is proposed. The experimental results on the UCSD Ped2, CUHK Avenue and ShanghaiTech datasets have obtained AUC values of 96.75%, 84.34% and 77.08% respectively, which shows that the proposed method attains competitive detection accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

We provide original and editable data appearing in the submitted article, including figures, tables and experimental results.

Code availability

We are pleased to share code that is used in work submitted for publication. Authors’ contributions: All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Hongjun Li, Jinyi Chen, Xiaohu Sun, Chaobo Li, and Junjie Chen. The first draft of the manuscript was written by Jinyi Chen and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

References

  1. Bahrami M, Pourahmadi M, Vafaei A, Shayesteh MR (2021) A comparative study between single and multi-frame anomaly detection and localization in recorded video streams. J Vis Commun Image Represent 79:1–10

    Article  Google Scholar 

  2. Bedja-Johnson Z, Wu P, Grande D, Anderlini E (2022) Smart anomaly detection for Slocum underwater gliders with a variational autoencoder with long short-term memory networks. Appl Ocean Res 120:1–14

    Article  Google Scholar 

  3. Bewley A, Ge ZY, Ott L, Ramos F, Upcroft B (2016) Simple online and realtime tracking. In: Processing of the 2016 IEEE international conference on image processing, pp 3464–3468

  4. Cai YH, Liu JQ, Guo YJ, Hu SB, Lang SN (2021) Video anomaly detection with multi-scale feature and temporal information fusion. Neurocomputing 423:264–273

    Article  Google Scholar 

  5. Chang YP, Tu ZG, Xie W, Yuan JS (2020) Clustering driven deep autoencoder for video anomaly detection. In: Processing of the computer vision–ECCV 2020: 16th European Conference, pp 329–345

  6. Chang YP, Tu ZG, Xie W, Luo B, Zhang SF, Sui HG, Yuan JS (2021) Video anomaly detection with spatio-temporal dissociation. Pattern Recogn 122:1–12

    Google Scholar 

  7. Chaudhary A, Tiwari VN, Kumar A (2014a) Design an anomaly based fuzzy intrusion detection system for packet dropping attack in mobile ad hoc networks. In: Processing of the 2014 IEEE international advance computing conference, pp 256–261

  8. Chaudhary A, Kumar A, Tiwari VN (2014b) A reliable solution against packet dropping attack due to malicious nodes using fuzzy logic in MANETs. In: Processing of the 2014 international conference on reliability optimization and information technology, pp 178–181

  9. Chaudhary A, Tiwari VN, Kumar A (2014c) A novel intrusion detection system for ad hoc flooding attack using fuzzy logic in mobile ad hoc networks. In: Processing of the international conference on recent advances and innovations in engineering, pp 1–4

  10. Chaudhary A, Tiwari VN, Kumar A (2015) A cooperative intrusion detection system for sleep deprivation attack using neuro-fuzzy classifier in Mobile ad hoc networks. Comput Intell Data Min 32:345–353

    Google Scholar 

  11. Chen DY, Wang PT, Yue LY, Zhang YX, Jia T (2020) Anomaly detection in surveillance video based on bidirectional prediction. Image Vis Comput 98:1–8

    Article  Google Scholar 

  12. Doshi K, Yilmaz Y (2021) Online anomaly detection in surveillance videos with asymptotic bound on false alarm rate. Pattern Recogn 114:1–9

    Article  Google Scholar 

  13. Fan YX, Wen GJ, Li D, Qiu SH, Levine MD, Xiao F (2020) Video anomaly detection and localization via Gaussian mixture fully convolutional Variational autoencoder. Comput Vis Image Underst 195:1–12

    Article  Google Scholar 

  14. Fernando T, Denman S, Ahmedt-Aristizabal D, Sridharan S, Laurens KR, Johnston P, Fookes C (2020) Neural memory plasticity for medical anomaly detection. Neural Netw 127:67–81

    Article  Google Scholar 

  15. Gong D, Liu LQ, Le V, Saha B, Mansour MR, Venkatesh S, Hengel AV (2019) Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection. IProc IEEE/CVF Conf Comput Vis Pattern Recognit:1705–1714

  16. Hao Y, Li J, Wang NN, Wang XY, Gao XB (2022) Spatiotemporal consistency-enhanced network for video anomaly detection. Pattern Recogn 121:1–11

    Article  Google Scholar 

  17. Hasan M, Choi J, Neumann J, Roy-Chowdhury AK, Davis LS (2016) Learning temporal regularity in video sequences. Proc IEEE Conf Comput Vis Pattern Recognit:733–742

  18. He KM, Zhang XY, Ren SQ, Sun J (2017) Simple online and Realtime tracking with a deep association metric. In: Processing of the 2017 IEEE international conference on image processing, pp 3645–3649

  19. Kingma DP, Welling M (2014) Auto-encoding variational bayes. In: Processing of the international conference on learning representations, pp 1–14

  20. Kumar K (2018) EVS-DK: Event Video Skimming using Deep Keyframe. J Vis Commun Image Represent 58:345–352

    Article  Google Scholar 

  21. Kumar K, Shrimankar DD (2018) Deep event learning boost-up approach: DELTA. Multimed Tools Appl 77(20):26635–26655

    Article  Google Scholar 

  22. Kumar K, Shrimankar DD (2018) F-DES: fast and deep event summarization. IEEE Trans Multimed 20(2):323–334

    Article  Google Scholar 

  23. Kumar K, Shrimankar DD, Singh N (2016) Equal partition based clustering approach for event summarization in videos. In: Processing of the 2016 12th international conference on signal-image Technology & Internet-Based Systems, pp 119–126

  24. Kumar K, Kumar A, Bahuguna A (2017a) D-CAD: deep and crowded anomaly detection. In: Processing of the 7th international conference on computer and communication technology, pp 100–105

  25. Kumar K, Shrimankar DD, Singh N (2017b) Event BAGGING: a novel event summarization approach in multiview surveillance videos. In: Processing of the 2017 international conference on innovations in electronics, signal processing and communication (IESC), pp 106–111

  26. Kumar K, Shrimankar DD, Singh N (2017) Eratosthenes sieve based key-frame extraction technique for event summarization in videos. Multimed Tools Appl 77(6):7383–7404

    Article  Google Scholar 

  27. Kumar K, Shrimankar DD, Singh N (2018) V-LESS: a video from linear event summaries. In: Processing of the proceedings of 2nd international conference on Computer Vision & Image Processing, pp 385–395

  28. Kumar K, Shrimankar DD, Singh N (2019) Key-lectures: Keyframes extraction in video lectures. Machine Intelligence and Signal Analysis:453–459

  29. Lee S, Kim HG, Choi DH, Kim H, Ro YM (2021) Video prediction recalling long-term motion context via memory alignment learning. Proc IEEE/CVF Conf Comput Vis Pattern Recognit:3054–3063

  30. Li WX, Mahadevan V, Vasconcelos N (2014) Anomaly detection and localization in crowded scenes. IEEE Trans Pattern Anal Mach Intell 36(1):18–32

  31. Li B, Leroux S, Simoens P (2021) Decoupled appearance and motion learning for efficient anomaly detection in surveillance video. Comput Vis Image Underst 210:1–8

    Article  Google Scholar 

  32. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollar P, Zitnick CL (2014) Microsoft COCO: common objects in context. In: Processing of the computer vision–ECCV 2014: 13th European conference, pp 740–755

  33. Lin TY, Goyal P, Girshick R, He K, Dollar P (2020) Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell 42:318–317

    Article  Google Scholar 

  34. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) SSD: single shot multibox detector. In: Processing of the computer vision–ECCV 2016: 14th European conference, pp 11–14

  35. Liu W, Luo WX, Lian DZ, Gao SH (2018) Future frame prediction for anomaly detection-a new baseline. Proc IEEE Conf Comput Vis Pattern Recogn:6536–6545

  36. Lu CW, Shi JP, Jia JY (2013) Abnormal event detection at 150 FPS in MATLAB. Proc IEEE Int Conf Comput Vis:2720–2727

  37. Luo WX, Liu W, Gao SH (2017) A revisit of sparse coding based anomaly detection in stacked RNN framework. Proc IEEE Int Conf Comput Vis:341–349

  38. Luo WX, Liu W, Gao SH (2021) Normal graph: spatial temporal graph convolutional networks based prediction network for skeleton based video anomaly detection. Neurocomputing 444:332–337

    Article  Google Scholar 

  39. Lv H, Chen C, Cui Z, Xu CY, Li Y, Yang J (2021) Learning normal dynamics in videos with meta prototype network. Proc IEEE/CVF Conf Comput Vis Pattern Recognit:15425–15434

  40. Park H, Noh J, Ham B (2020) Learning memory-guided normality for anomaly detection. In: IEEE/CVF Conf. Comput Vis Pattern Recognit, 14360–14369

  41. Redmon J, Farhadi A (2017) YOLO9000: Better, faster, stronger. Proc IEEE Conf Comput Vis Pattern Recognit:7263–7271

  42. Ren SQ, He KM, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Conf. Neural Inf. Process. Syst. 28:91–99

  43. Shin W, Bu SJ, Cho SB (2020) 3D-convolutional neural network with generative adversarial network and autoencoder for robust anomaly detection in video surveillance. Int J Neural Syst 30:1–15

    Article  Google Scholar 

  44. Sun P, Zhang RF, Jiang Y, Kong T, Xu CF, Zhan W, Tomizuka M, Li L, Yuan ZH, Wang CH, Luo P (2021) Sparse R-CNN: end-to-end object detection with learnable proposals. Proc IEEE/CVF Conf Comput Vis Pattern Recognit:14454–14463

  45. Wang ZG, Zhang YJ, Wang GJ, Xie PW (2021) Main-auxiliary aggregation strategy for video anomaly detection. IEEE Signal Process Lett 28:1794–1798

    Article  Google Scholar 

  46. Wang WQ, Chang F, Mi HD (2021) Intermediate fused network with multiple timescales for anomaly detection. Neurocomputing 433:37–49

    Article  Google Scholar 

  47. Wei BB, Chen HY, Ding QH, Luo HB (2022) SiamOAN: Siamese object-aware network for real-time target tracking. Neurocomputing 471:161–174

    Article  Google Scholar 

  48. Wojke N, Bewley A, Paulus D (2017) Simple online and Realtime tracking with a deep association metric. In: Processing of the 2017 IEEE international conference on image processing, pp 3645–3649

  49. Wu RZ, Li S, Chen CLZ, Hao AM (2021) Improving video anomaly detection performance by mining useful data from unseen video frames. Neurocomputing 462:523–553

    Article  Google Scholar 

  50. Wu CK, Shao S, Tunc C, Satam P, Hariri S (2021) An explainable and efficient deep learning framework for video anomaly detection. Clust Comput https://doi.org/10.1007/s10586-021-03439-5

  51. Xu Z, Zeng XQ, Ji GL, Sheng B (2021) Improved anomaly detection in surveillance videos with multiple probabilistic models inference. Intell Autom Soft Comput 31:1703–1717

    Article  Google Scholar 

  52. Yu L, Qiao BJ, Zhang HL, Yu JY, He X (2022) LTST: long-term segmentation tracker with memory attention network. Image Vis Comput 118:1–10

    Google Scholar 

  53. Zhong YH, Chen X, Jiang JY, Ren F (2020) A cascade reconstruction model with generalization ability evaluation for anomaly detection in videos. Pattern Recogn 122:108336

    Article  Google Scholar 

Download references

Funding

This work is supported in part by National Natural Science Foundation of China under Grant 61871241, Grant 61971245 and Grant 61976120, in part by Nanjing University State Key Lab. for Novel Software Technology under Grant KFKT2019B15, in part by Nantong Science and Technology Program JC2021131 and in part by Postgraduate Research and Practice Innovation Program of Jiangsu Province KYCX21_3084 and KYCX22_3340.

Author information

Authors and Affiliations

  1. School of Information Science and Technology, Nantong University, 9 Seyuan road, Nantong, 226019, Jiangsu, People’s Republic of China

    Hongjun Li, Jinyi Chen, Xiaohu Sun, Chaobo Li & Junjie Chen

  2. State Key Lab. for Novel Software Technology, Nanjing University, Nanjing, 210023, Jiangsu, People’s Republic of China

    Hongjun Li

  3. Nantong Research Institute for Advanced Communication Technologies, Nantong, 226019, Jiangsu, People’s Republic of China

    Hongjun Li & Junjie Chen

  4. TONGKE School of Microelectronics, Nantong, 226019, Jiangsu, People’s Republic of China

    Hongjun Li & Junjie Chen

Authors
  1. Hongjun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinyi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaohu Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chaobo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Junjie Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongjun Li.

Ethics declarations

Conflicts of interest/competing interests

None.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Chen, J., Sun, X. et al. Multi-memory video anomaly detection based on scene object distribution. Multimed Tools Appl 82, 35557–35583 (2023). https://doi.org/10.1007/s11042-023-14956-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14956-3

Keywords

Search

Navigation