Skip to main content
SpringerLink
Log in

Domain generalization for video anomaly detection considering diverse anomaly types

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

In intelligent video surveillance, anomaly detection is conducted to identify the occurrence of abnormal events by monitoring the video captured by vision sensors, and it has important application value in public safety, industrial production process monitoring, and other fields. However, the study of video anomaly detection (VAD) models that generalize into uncharted territory remains challenging. The current approach is to use more abnormal samples to enhance the training and improve the generalization ability of the model, but this requires a large number of auxiliary datasets to fully describe the abnormal events. In addition, the ambiguity of the abnormality definition makes it impossible to effectively cover all abnormal videos. To solve this problem, we divide abnormalities into three different types based on object and behavior type: normal object and abnormal behavior (NOAB), abnormal object and normal behavior (AONB), and abnormal object and abnormal behavior (AOAB). We find that the traditional prediction-based model shows better generalization ability for NOAB. However, under the domain generalization setting, the generalization effect of AONB and AOAB decreases significantly. To solve this problem, a new spatiotemporal generalization (STG) model is proposed specifically for the detection of AONB and AOAB events involving anomalous objects. The STG model integrates contrast learning and adaptive data augmentation techniques to realize domain expansion. In addition, combining the STG model with the traditional prediction-based model, an anomaly video monitoring framework is proposed to comprehensively detect anomalies without target domain adaptation, which can improve the generalization ability of VAD models without auxiliary datasets. Extensive evaluations show that the proposed method achieves excellent performance on the benchmark datasets under the domain generalization setting.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The datasets used or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Liu, W., Luo, W., Lian, D., Gao, S.: Future frame prediction for anomaly detection–a new baseline. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6536–6545 (2018)

  2. Chen, H., Mei, X., Ma, Z., Wu, X., Wei, Y.: Spatial-temporal graph attention network for video anomaly detection. Image Vis. Comput. 131, 104629 (2023)

    Article  Google Scholar 

  3. Chen, D., Wang, P., Yue, L., Zhang, Y., Jia, T.: Anomaly detection in surveillance video based on bidirectional prediction. Image Vis. Comput. 98, 103915 (2020)

    Article  Google Scholar 

  4. Ramachandra, B., Jones, M.: Street scene: A new dataset and evaluation protocol for video anomaly detection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2569–2578 (2020)

  5. Dong, F., Zhang, Y., Nie, X.: Dual discriminator generative adversarial network for video anomaly detection. IEEE Access 8, 88170–88176 (2020)

    Article  Google Scholar 

  6. Chandrakala, S., Shalmiya, P., Srinivas, V., Deepak, K.: Object-centric and memory-guided network-based normality modeling for video anomaly detection. SIViP 16(7), 2001–2007 (2022)

    Article  Google Scholar 

  7. Xu, H., Liu, W., Xing, W., Wei, X.: Motion-aware future frame prediction for video anomaly detection based on saliency perception. SIViP 16(8), 2121–2129 (2022)

    Article  Google Scholar 

  8. Deepak, K., Chandrakala, S., Mohan, C.K.: Residual spatiotemporal autoencoder for unsupervised video anomaly detection. SIViP 15(1), 215–222 (2021)

    Article  Google Scholar 

  9. Xu, Q., Zhang, R., Wu, Y.-Y., Zhang, Y., Liu, N., Wang, Y.: Simde: A simple domain expansion approach for single-source domain generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4797–4807 (2023)

  10. Li, Y., Tian, X., Gong, M., Liu, Y., Liu, T., Zhang, K., Tao, D.: Deep domain generalization via conditional invariant adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 624–639 (2018)

  11. Matsuura, T., Harada, T.: Domain generalization using a mixture of multiple latent domains. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 11749–11756 (2020)

  12. Yao, Z., Wang, Y., Wang, J., Philip, S.Y., Long, M.: Videodg: Generalizing temporal relations in videos to novel domains. IEEE Trans. Pattern Anal. Mach. Intell. 44(11), 7989–8004 (2021)

    Article  Google Scholar 

  13. Volpi, R., Namkoong, H., Sener, O., Duchi, J.C., Murino, V., Savarese, S.: Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems, vol. 31 (2018)

  14. Lu, Y., Yu, F., Reddy, M.K.K., Wang, Y.: Few-shot scene-adaptive anomaly detection. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pp. 125–141. Springer (2020)

  15. Lv, H., Chen, C., Cui, Z., Xu, C., Li, Y., Yang, J.: Learning normal dynamics in videos with meta prototype network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15425–15434 (2021)

  16. Roy, P.R., Bilodeau, G.-A., Seoud, L.: Predicting next local appearance for video anomaly detection. In: 2021 17th International Conference on Machine Vision and Applications (MVA), pp. 1–5. IEEE (2021)

  17. Aich, A., Peng, K.-C., Roy-Chowdhury, A.K.: Cross-domain video anomaly detection without target domain adaptation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2579–2591 (2023)

  18. Wu, S., Moore, B.E., Shah, M.: Chaotic invariants of lagrangian particle trajectories for anomaly detection in crowded scenes. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2054–2060. IEEE (2010)

  19. Tung, F., Zelek, J.S., Clausi, D.A.: Goal-based trajectory analysis for unusual behaviour detection in intelligent surveillance. Image Vis. Comput. 29(4), 230–240 (2011)

    Article  Google Scholar 

  20. Adam, A., Rivlin, E., Shimshoni, I., Reinitz, D.: Robust real-time unusual event detection using multiple fixed-location monitors. IEEE Trans. Pattern Anal. Mach. Intell. 30(3), 555–560 (2008)

    Article  Google Scholar 

  21. Kim, J., Grauman, K.: Observe locally, infer globally: a space-time mrf for detecting abnormal activities with incremental updates. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2928. IEEE (2009)

  22. Xu, D., Ricci, E., Yan, Y., Song, J., Sebe, N.: Learning deep representations of appearance and motion for anomalous event detection. arXiv preprint arXiv:1510.01553 (2015)

  23. Liu, Z., Nie, Y., Long, C., Zhang, Q., Li, G.: A hybrid video anomaly detection framework via memory-augmented flow reconstruction and flow-guided frame prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13588–13597 (2021)

  24. Huang, C., Wen, J., Xu, Y., Jiang, Q., Yang, J., Wang, Y., Zhang, D.: Self-supervised attentive generative adversarial networks for video anomaly detection. IEEE Trans. Neural Netw. Learn. Syst. (2022)

  25. Roy, P.R., Bilodeau, G.-A., Seoud, L.: Predicting next local appearance for video anomaly detection. In: 2021 17th International Conference on Machine Vision and Applications (MVA), pp. 1–5. IEEE (2021)

  26. Georgescu, M.I., Ionescu, R.T., Khan, F.S., Popescu, M., Shah, M.: A background-agnostic framework with adversarial training for abnormal event detection in video. IEEE Trans. Pattern Anal. Mach. Intell. 44(9), 4505–4523 (2021)

    Google Scholar 

  27. Doshi, K., Yilmaz, Y.: Any-shot sequential anomaly detection in surveillance videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 934–935 (2020)

  28. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234–241. Springer (2015)

  29. Chen, X., He, K.: Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750–15758 (2021)

  30. Luo, W., Liu, W., Gao, S.: A revisit of sparse coding based anomaly detection in stacked rnn framework. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)

  31. Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6479–6488 (2018)

  32. Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 18–32 (2013)

    Google Scholar 

  33. Lu, C., Shi, J., Jia, J.: Abnormal event detection at 150 fps in matlab. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2720–2727 (2013)

  34. Yu, G., Wang, S., Cai, Z., Zhu, E., Xu, C., Yin, J., Kloft, M.: Cloze test helps: Effective video anomaly detection via learning to complete video events. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 583–591 (2020)

  35. Cai, Z., Vasconcelos, N.: Cascade r-cnn: Delving into high quality object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6154–6162 (2018)

  36. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: Evolution of optical flow estimation with deep networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2462–2470 (2017)

  37. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017)

  38. Kinga, D., Adam, J.B., : A method for stochastic optimization. In: International Conference on Learning Representations (ICLR), vol. 5, p. 6 . San Diego, California (2015)

Download references

Funding

This work is supported by the National Natural Science Foundation of China under Grant No. 61973122 (Xiaojing Gu) and 61973120 (Xingsheng Gu)

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China

    Zhiqiang Wang, Xiaojing Gu, Huaicheng Yan & Xingsheng Gu

Authors
  1. Zhiqiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaojing Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huaicheng Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xingsheng Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Zhiqiang Wang was involved in conceptualization, methodology, software, and writing—original draft. Xiaojing Gu was responsible for conceptualization, methodology, analysis, writing—reviewing and editing, supervision, and funding acquisition. Huaicheng Yan contributed to data curation and writing— reviewing and editing. Xingsheng Gu took part in writing— reviewing and editing, supervision, and funding acquisition.

Corresponding authors

Correspondence to Xiaojing Gu or Xingsheng Gu.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Z., Gu, X., Yan, H. et al. Domain generalization for video anomaly detection considering diverse anomaly types. SIViP 18, 3691–3704 (2024). https://doi.org/10.1007/s11760-024-03033-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-024-03033-3

Keywords

Search

Navigation