Skip to main content
SpringerLink
Log in

Future frame prediction based on generative assistant discriminative network for anomaly detection

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Anomaly detection plays an important role in intelligent surveillance and has attracted increasing attention from researchers in recent years. It is generally regarded as discrimination that cannot be properly represented in most approaches. Despite its importance in probing the scarcity and indefinability of abnormal data during training, designing an effective network is exceptionally complex due to the diversity of the motion information, difficulty of parsing prediction errors, robustness and so on. To improve the ability to extract subtle features between normal and abnormal frames, and to improve the robustness to noise, a generative assistant discriminative network for anomaly detection is proposed. This method detects anomalies by predicting future frames in combination of adversary and cooperation, which mainly consists of the generator, discriminator and assistor. The generator predicts the future frames, while the discriminator distinguishes the predicted future frames from actual future frames. Moreover, by means of noise, the assistor is able to learn from pseudo abnormal future frames and predicted future frames. This helps the generator strengthen the ability to extract the discriminative features between normal and abnormal events. The motion information is used for integration into the predicted future frames. Extensive experiments are conducted on the UCSD Ped1, Ped2, CUHK Avenue and ShanghaiTech datasets. A comparison with the state-of-the-art methods shows the effectiveness and advantages of our method for anomaly detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Code Availability

We are pleased to share code that is used in work submitted for publication.

References

  1. Afiq AA, Zakariya MN, Saad MN, Nurfarzana AA, Khir MHM, Fadzil AF, Jale A, Witjaksono G, Zainal-Abidin I, Faizari M (2018) A review on classifying abnormal behavior in crowd scene. J Vis Commun Image Represent 58:285–303

    Article  Google Scholar 

  2. Chen G, Liu P, Liu Z, Tang H, Hong L, Dong J, Conradt J, Knoll A (2021) Neuroaed: Towards efficient abnormal event detection in visual surveillance with neuromorphic vision sensor. IEEE Transactions on Information Forensics and Security 16:923–936

    Article  Google Scholar 

  3. Socha R, Kogut B (2020) Urban video surveillance as a tool to improve security in public spaces. Sustainability 12(15):6210

    Article  Google Scholar 

  4. Ammar S, Bouwmans T, Zaghden N, Neji M (2020) Deep detector classifier (deepdc) for moving objects segmentation and classification in video surveillance. IET Image Process 14(8):1490–1501

    Article  Google Scholar 

  5. Yu Q, Luo Y, Chen C, Wang X (2018) Trajectory outlier detection approach based on common slices sub-sequence. Appl Intell 48(9):2661–2680

    Article  Google Scholar 

  6. Luo W, Liu W, Lian D, Tang J, Duan L, Peng X, Gao S (2021) Video anomaly detection with sparse coding inspired deep neural networks. IEEE Trans Pattern Anal Mach Intell 43(3):1070–1084

    Article  Google Scholar 

  7. Lin D, Li Y, Xie S, Nwe TL, Dong S (2021) Ddr-id: Dual deep reconstruction networks based image decomposition for anomaly detection. Journal of Ambient Intelligence and Humanized Computing, pp 1–15

  8. Pang G, Shen C, Cao L, Hengel AVD (2021) Deep learning for anomaly detection: A review. ACM Comput Surv 54(2):1–38

    Article  Google Scholar 

  9. Guo J, Zheng P, Huang J (2019) Efficient privacy-preserving anomaly detection and localization in bitstream video. IEEE Trans Circuits Syst Video Technol 30(9):3268–3281

    Article  Google Scholar 

  10. Murugan BS, Elhoseny M, Shankar K, Uthayakumar J (2019) Region-based scalable smart system for anomaly detection in pedestrian walkways. Comput Electr Eng 75:146–160

    Article  Google Scholar 

  11. Ji QG, Chi R, Lu ZM (2018) Anomaly detection and localization in the crowd scenes using a block-based social force model (bsfm). Iet Image Process 12(1):133–137

    Article  Google Scholar 

  12. Li W, Mahadevan V, Vasconcelos N (2013) Anomaly detection and localization in crowded scenes. IEEE Trans Pattern Anal Mach Intell 36(1):18–32

    Google Scholar 

  13. Luo Z, He K, Yu Z (2021) A robust unsupervised anomaly detection framework. Appl Intell 2:1–15

    Google Scholar 

  14. Canizo M, Conde AIT, Onieva E (2019) Multi-head cnn–rnn for multi-time series anomaly detection: An industrial case study. Neurocomputing 363:246–260

    Article  Google Scholar 

  15. Bansod S, Nandedkar A (2019) Transfer learning for video anomaly detection. Journal of Intelligent and Fuzzy Systems 36(3):1967–1975

    Article  Google Scholar 

  16. Jiang T, Li Y, Xie W, Du Q (2020) Discriminative reconstruction constrained generative adversarial network for hyperspectral anomaly detection. IEEE Trans Geosci Remote Sens 58(7):4666–4679

    Article  Google Scholar 

  17. Yan Y, Wang D, Zhou G, Chen Q (2021) Unsupervised anomaly segmentation via multilevel image reconstruction and adaptive attention-level transition. IEEE Trans Instrum Meas 70:1–12

    Google Scholar 

  18. Tang Y, Zhao L, Zhang S, Gong C, Li G, Yang J (2020) Integrating prediction and reconstruction for anomaly detection. Pattern Recogn Lett 129:123–130

    Article  Google Scholar 

  19. Chen D, Wang P, Yue L, Zhang Y, Jia T (2020) Anomaly detection in surveillance video based on bidirectional prediction. Image Vis Comput 98:103915

    Article  Google Scholar 

  20. Zhou JT, Zhang L, Fang Z, Du J, Peng X, Xiao Y (2019) Attention-driven loss for anomaly detection in video surveillance. IEEE Trans Circuits Syst Video Technol 30(12):4639–4647

    Article  Google Scholar 

  21. Sabokrou M, Fayyaz M, Fathy M, Moayed Z, Klette R (2018) Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes. Comput Vis Image Underst 172:88–97

    Article  MATH  Google Scholar 

  22. Sabokrou M, Fayyaz M, Fathy M, Klette R (2017) Deep-cascade: Cascading 3d deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans Image Process 26(4):1992–2004

    Article  MathSciNet  MATH  Google Scholar 

  23. Xu K, Sun T, Jiang X (2019) Video anomaly detection and localization based on an adaptive intra-frame classification network. IEEE Transactions on Multimedia 22(2):394–406

    Article  Google Scholar 

  24. Gong M, Zeng H, Xie Y, Li H, Tang Z (2020) Local distinguishability aggrandizing network for human anomaly detection. Neural Netw 122:364–373

    Article  Google Scholar 

  25. Fan Y, Wen G, Li D, Qiu S, Levine MD, Xiao F (2020) Video anomaly detection and localization via gaussian mixture fully convolutional variational autoencoder. Comput Vis Image Underst 195:102920

    Article  Google Scholar 

  26. Lei J, Fang S, Xie W, Li Y, Chang C-I (2020) Discriminative reconstruction for hyperspectral anomaly detection with spectral learning. IEEE Trans Geosci Remote Sens 58(10):7406–7417

    Article  Google Scholar 

  27. Sabokrou M, Fathy M, Zhao G, Adeli E (2020) Deep end-to-end one-class classifier. IEEE Trans Neural Netw Learn Syst 32(2):675–684

    Article  Google Scholar 

  28. Chen D, Yue L, Chang X, Xu M, Jia T (2021) Nm-gan: Noise-modulated generative adversarial network for video anomaly detection. Pattern Recogn 116:107969

    Article  Google Scholar 

  29. Li N, Chang F, Liu C (2020) Spatial-temporal cascade autoencoder for video anomaly detection in crowded scenes. IEEE Transactions on Multimedia 23:203–215

    Article  Google Scholar 

  30. Song H, Sun C, Wu X, Chen M, Jia Y (2019) Learning normal patterns via adversarial attention-based autoencoder for abnormal event detection in videos. IEEE Transactions on Multimedia 22(8):2138–2148

    Article  Google Scholar 

  31. Liu W, Luo W, Lian D, Gao S (2018) Future frame prediction for anomaly detection–a new baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6536–6545

  32. Lu Y, Kumar KM, shahabeddin Nabavi S, Wang Y (2019) Future frame prediction using convolutional vrnn for anomaly detection. In: 2019 16Th IEEE international conference on advanced video and signal based surveillance (AVSS), pp 1–8

  33. Lee S, Kim HG, Ro YM (2019) Bman: Bidirectional multi-scale aggregation networks for abnormal event detection. IEEE Trans Image Process 29:2395–2408

    Article  MATH  Google Scholar 

  34. Wang C, Xu C, Yao X, Tao D (2019) Evolutionary generative adversarial networks. IEEE Trans Evol Comput 23(6):921–934

    Article  Google Scholar 

  35. Han J, Tao J, Wang C (2018) Flownet: A deep learning framework for clustering and selection of streamlines and stream surfaces. IEEE Trans Vis Comput Graph 26(4):1732–1744

    Google Scholar 

  36. Hansen L, Heinrich MP (2021) Graphregnet: Deep graph regularisation networks on sparse keypoints for dense registration of 3d lung cts. IEEE Trans Med Imaging 40(9):2246–2257

    Article  Google Scholar 

  37. Sikdar A, Chowdhury AS (2020) An adaptive training-less framework for anomaly detection in crowd scenes. Neurocomputing 415:317–331

    Article  Google Scholar 

  38. Wang X, Che Z, Jiang B, Xiao N, Yang K, Tang J, Ye J, Wang J, Qi Q (2021) Robust unsupervised video anomaly detection by multipath frame prediction. IEEE Transactions on Neural Networks and Learning Systems, pp 1–12

  39. Colque RVHM, Caetano C, de Andrade MTL, Schwartz WR (2016) Histograms of optical flow orientation and magnitude and entropy to detect anomalous events in videos. IEEE Trans Circuits Syst Video Technol 27(3):673–682

    Article  Google Scholar 

  40. Tudor Ionescu R, Smeureanu S, Alexe B, Popescu M (2017) Unmasking the abnormal events in video. In: Proceedings of the IEEE international conference on computer vision, pp 2895–2903

  41. Chu W, Xue H, Yao C, Cai D (2018) Sparse coding guided spatiotemporal feature learning for abnormal event detection in large videos. IEEE Transactions on Multimedia 21(1):246–255

    Article  Google Scholar 

  42. Ravanbakhsh M, Sangineto E, Nabi M, Sebe N (2019) Training adversarial discriminators for cross-channel abnormal event detection in crowds. In: 2019 IEEE Winter conference on applications of computer vision, pp 1896–1904

  43. Khan MA, Kim J (2020) Toward developing efficient conv-ae-based intrusion detection system using heterogeneous dataset. Electronics 9(11):1771

    Article  Google Scholar 

  44. Zhou JT, Du J, Zhu H, Peng X, Liu Y, Goh RSM (2019) Anomalynet: An anomaly detection network for video surveillance. IEEE Transactions on Information Forensics and Security 14(10):2537–2550

    Article  Google Scholar 

  45. Gong D, Liu L, Le V, Saha B, Mansour MR, Venkatesh S, Hengel AVD (2019) Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In: IEEE International conference on computer vision, pp 1705–1714

  46. Yan S, Smith JS, Lu W, Zhang B (2018) Abnormal event detection from videos using a two-stream recurrent variational autoencoder. IEEE Transactions on Cognitive and Developmental Systems 12 (1):30–42

    Article  Google Scholar 

  47. Salehi M, Arya A, Pajoum B, Otoofi M, Shaeiri A, Rohban MH, Rabiee HR (2021) Arae: Adversarially robust training of autoencoders improves novelty detection. Neural Netw 144:726–736

    Article  Google Scholar 

  48. Park H, Noh J, Ham B (2020) Learning memory-guided normality for anomaly detection. In: IEEE Conference on computer vision and pattern recognition, pp 14372–14381

  49. Li S, Fang J, Xu H, Xue J (2021) Video frame prediction by deep multi-branch mask network. IEEE Trans Circuits Syst Video Technol 31(4):1283–1295

    Article  Google Scholar 

  50. Luo W, Liu W, Lian D, Gao S (2021) Future frame prediction network for video anomaly detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, in press

  51. Zhang Q, Feng G, Wu H (2022) Surveillance video anomaly detection via non-local u-net frame prediction. Multimedia Tools and Applications, in press

Download references

Acknowledgements

This work is supported in part by National Natural Science Foundation of China under Grant 61871241, Grant 61971245 and Grant 61976120, in part by Nanjing University State Key Lab. for Novel Software Technology under Grant KFKT2019B15, in part by the Nantong Science and Technology Program of JC2021131, and in part by Postgraduate Research and Practice Innovation Program of Jiangsu Province under Grant KYCX21_3084.

Funding

This work is supported in part by National Natural Science Foundation of China under Grant 61871241, Grant 61971245 and Grant 61976120, in part by Nanjing University State Key Lab. for Novel Software Technology under Grant KFKT2019B15, in part by the Nantong Science and Technology Program of JC2021131, and in part by Postgraduate Research and Practice Innovation Program of Jiangsu Province under Grant KYCX21_3084.

Author information

Authors and Affiliations

  1. School of Information Science and Technology, Nantong University, 9 Seyuan Road, Nantong, 226019, China

    Chaobo Li, Hongjun Li & Guoan Zhang

  2. State Key Lab. for Novel Software Technology, Nanjing University, 163 Xianlin Avenue, Nanjing, 210023, China

    Hongjun Li

Authors
  1. Chaobo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongjun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guoan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Chaobo Li, Hongjun Li, Guoan Zhang. The first draft of the manuscript was written by Chaobo Li and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Hongjun Li.

Ethics declarations

Conflict of Interests

We declare that no competing interests.

Additional information

Availability of data and materials

We provide original and editable data appearing in the submitted article, including figures, tables and experimental results.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, C., Li, H. & Zhang, G. Future frame prediction based on generative assistant discriminative network for anomaly detection. Appl Intell 53, 542–559 (2023). https://doi.org/10.1007/s10489-022-03488-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03488-2

Keywords

Search

Navigation