Skip to main content
SpringerLink
Log in

Abnormal behavior detection using streak flow acceleration

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The goal of abnormal behavior detection is to detect an anomalous event in video as accurate as possible. Motion information is crucial in such case as an inadequate motion estimation can easily make it worse. In this work, an abnormal event detection method was proposed to detect the occurrence of an anomaly automatically by using generative adversarial network (GAN) and streak flow acceleration. The proposed method is mainly composed of two components: (1) GAN-based framework that feeds on motion patterns to detect abnormal events, and (2) explicitly modeling motion information by incorporating streak flow acceleration. The effectiveness of the proposed model is verified on public benchmarks and comparative results show that our method performs favorably against many state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Akcay S, Atapour-Abarghouei A, Breckon TP (2019) GANomaly: Semi-supervised Anomaly Detection via Adversarial Training. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol 11363 LNCS, pp 622–637. https://doi.org/10.1007/978-3-030-20893-6_39

  2. Bastan M, Yilmaz Ö (2016) Multi-view Product Image Search Using ConvNets Features. CoRR

  3. Ben Mabrouk A, Zagrouba E (2017) Spatio-temporal feature using optical flow based distribution for violence detection. Pattern Recogn Lett 92:62–67. https://doi.org/10.1016/j.patrec.2017.04.015

    Article  Google Scholar 

  4. Bilinski P, Bremond F (2016) Human violence recognition and detection in surveillance videos. In: 2016 13Th IEEE international conference on advanced video and signal based surveillance, AVSS 2016, pp 30–36. https://doi.org/10.1109/AVSS.2016.7738019

  5. Bird N, Atev S, Caramelli N, Martin R, Masoud O, Papanikolopoulos N (2006) Real time, online detection of abandoned objects in public areas. In: Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006. IEEE, pp 3775–3780

  6. Cheng KW, Chen YT, Fang WH (2015) Gaussian process regression-based video anomaly detection and localization with hierarchical feature representation. IEEE Trans Image Process 24(12):5288–5301

    Article  MathSciNet  Google Scholar 

  7. Christoudias CM, Urtasun R, Darrell T (2008) Unsupervised feature selection via distributed coding for multi-view object recognition. In: 26Th IEEE conference on computer vision and pattern recognition, CVPR. IEEE, pp 1–8. https://doi.org/10.1109/CVPR.2008.4587615

  8. Colque RVHM, Caetano C, de Andrade MTL, Schwartz WR (2016) Histograms of optical flow orientation and magnitude and entropy to detect anomalous events in videos. IEEE Trans Circ Syst Video Technol 27(3):673–682

    Article  Google Scholar 

  9. Cui S, Wang S, Zhuo J, Li L, Huang Q, Tian Q (2020) Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3941–3950

  10. Del Giorno A, Andrew Bagnell J, Hebert M (2016) A discriminative framework for anomaly detection in large videos. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol 9909 LNCS, pp 334–349. https://doi.org/10.1007/978-3-319-46454-1_21

  11. Diba A, Sharma V, Van Gool L (2017) Deep temporal linear encoding networks. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp 2329–2338

  12. Edison A, Jiji C (2015) Hsga: a novel acceleration descriptor for human action recognition. In: 2015 Fifth national conference on computer vision, pattern recognition, image processing and graphics (NCVPRIPG). IEEE, pp 1–4

  13. Edison A, Jiji C (2017) Optical acceleration for motion description in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 39–47

  14. Edison A, Jiji C (2019) Automated video analysis for action recognition using descriptors derived from optical acceleration. SIViP 13(5):915–922

    Article  Google Scholar 

  15. Eldar A Dense optical flow acceleration (2018). US Patent 10,074,151

  16. Feichtenhofer C, Pinz A, Zisserman A (2016) Convolutional two-stream network fusion for video action recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1933–1941. https://doi.org/10.1109/CVPR.2016.213. http://www.researchgate.net/publication/301878129

  17. Gao M, Jiang J, Ma L, Zhou S, Zou G, Pan J, Liu Z (2019) Violent crowd behavior detection using deep learning and compressive sensing. In: 2019 Chinese control and decision conference (CCDC). IEEE, pp 5329–5333

  18. Gao M, Jiang J, Shen J, Zou G, Fu G (2018) Crowd motion segmentation and behavior recognition fusing streak flow and collectiveness. Opt Eng 57(04):1. https://doi.org/10.1117/1.oe.57.4.043109

    Article  Google Scholar 

  19. Gao Y, Liu H, Sun X, Wang C, Liu Y (2016) Violence detection using Oriented VIolent Flows. Image Vis Comput 48-49:37–41. https://doi.org/10.1016/j.imavis.2016.01.006. http://www.sciencedirect.com/science/article/pii/S0262885616300063

  20. George M, Jose BR, Mathew J, Kokare P (2019) Autoencoder-based abnormal activity detection using parallelepiped spatio-temporal region. IET Comput Vis 13(1):23–30. https://doi.org/10.1049/iet-cvi.2018.5240. http://www.researchgate.net/publication/327509358

  21. Gong D, Liu L, Le V, Saha B, Mansour MR, Venkatesh S, Hengel A (2020) Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In: 2019 IEEE/CVF International conference on computer vision (ICCV)

  22. Gong D, Liu L, Le V, Saha B, Mansour MR, Venkatesh S, Hengel Avd (2019) Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

  23. Hasan M, Choi J, Neumann J, Roy-Chowdhury AK, Davis LS (2016) Learning temporal regularity in video sequences. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 733–742. https://doi.org/10.1109/CVPR.2016.86

  24. Hassner T, Itcher Y, Kliper-Gross O (2012) Violent flows: Real-time detection of violent crowd behavior. In: 2012 IEEE Computer society conference on computer vision and pattern recognition workshops. IEEE, pp 1–6

  25. Hassner T, Itcher Y, Kliper-Gross O (2012) Violent flows: Real-time detection of violent crowd behavior. In: IEEE Computer society conference on computer vision and pattern recognition workshops, pp 1–6. https://doi.org/10.1109/CVPRW.2012.6239348

  26. Hinami R, Mei T, Satoh S (2017) Joint detection and recounting of abnormal events by learning deep generic knowledge. In: Proceedings of the IEEE International Conference on Computer Vision, pp 3619–3627

  27. Huchuan Z, Ying R, Xiang S, Shun L (2016) Video anomaly detection based on locality sensitive hashing filters. Pattern Recogn J Pattern Recogn Soc 59:302–311

    Article  Google Scholar 

  28. Ionescu RT, Khan FS, Georgescu MI, Shao L (2019) Object-centric auto-encoders and dummy anomalies for abnormal event detection in video. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 7834–7843. https://doi.org/10.1109/CVPR.2019.00803

  29. Ionescu RT, Smeureanu S, Alexe B, Popescu M (2017) Unmasking the Abnormal Events in Video. Proceedings of the IEEE International Conference on Computer Vision, 2914–2922. https://doi.org/10.1109/ICCV.2017.315. 1705.08182

  30. Kataoka H, He Y, Shirakabe S, Satoh Y (2016) Motion representation with acceleration images. In: European conference on computer vision. Springer, pp 18–24

  31. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv:1412.6980

  32. Kiran BR, Thomas DM, Parakkal R (2018) An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. https://doi.org/10.3390/jimaging4020036

  33. Kratz L, Nishino K (2009) Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In: 2009 IEEE Conference on computer vision and pattern recognition. IEEE, pp 1446–1453

  34. Lee S, Kim HG, Ro YM (2020) BMAN: Bidirectional Multi-Scale Aggregation networks for abnormal event detection. IEEE Trans Image Process 29:2395–2408

    Article  Google Scholar 

  35. Leyva R, Sanchez V, Li CT (2017) Video Anomaly Detection With Compact Feature Sets for Online Performance. IEEE Trans Image Process 26(7), 3463–3478. https://doi.org/10.1109/TIP.2017.2695105. http://www.ncbi.nlm.nih.gov/pubmed/28436865

  36. Li A, Miao Z, Cen Y (2017) Global anomaly detection in crowded scenes based on optical flow saliency. In: 2016 IEEE 18Th international workshop on multimedia signal processing, MMSP 2016. https://doi.org/10.1109/MMSP.2016.7813390

  37. Li C, Han Z, Ye Q, Jiao J (2011) Abnormal behavior detection via sparse reconstruction analysis of trajectory. In: Proceedings - 6th International Conference on Image and Graphics, ICIG 2011, pp 807–810. https://doi.org/10.1109/ICIG.2011.104. http://www.researchgate.net/publication/229033791

  38. Liu W, Luo W, Lian D, Gao S (2018) Future frame prediction for anomaly detection–a new baseline. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 6536–6545

  39. Liu Y, Li CL, Póczos B (2018) Classifier two sample test for video anomaly detections. In: BMVC, pp 71

  40. Lu C, Shi J, Wang W, Jia J (2019) Fast abnormal event detection. Int J Comput Vis 127(8):993–1011. https://doi.org/10.1007/s11263-018-1129-8

    Article  Google Scholar 

  41. Lucas BD, Kanade T (1997) An iterative image registration technique with an application tostereo vision. In: Proceedings of the 7th International Joint Conference on ArtificialIntelligence

  42. Luo W, Liu W, Gao S (2017) A revisit of sparse coding based anomaly detection in stacked rnn framework. In: Proceedings of the IEEE International Conference on Computer Vision, pp 341–349

  43. Luo W, Liu W, Lian D, Tang J, Duan L, Peng X, Gao S (2021) Video anomaly detection with sparse coding inspired deep neural networks. IEEE Trans Pattern Anal Mach Intell 43(3):1070–1084. https://doi.org/10.1109/TPAMI.2019.2944377

    Article  Google Scholar 

  44. Mabrouk AB, Zagrouba E (2017) Spatio-temporal feature using optical flow based distribution for violence detection. Pattern Recogn Lett 92:62–67

    Article  Google Scholar 

  45. Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. In: 2010 IEEE Computer society conference on computer vision and pattern recognition. IEEE, pp 1975–1981

  46. Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 1975–1981. https://doi.org/10.1109/CVPR.2010.5539872

  47. Mehran R, Moore BE, Shah M (2010) A streakline representation of flow in crowded scenes. In: European conference on computer vision. Springer, pp 439–452

  48. Memisevic R (2012) On multi-view feature learning. arXiv:1206.4609

  49. Nallaivarothayan H, Fookes C, Denman S, Sridharan S (2014) An mrf based abnormal event detection approach using motion and appearance features. In: 2014 11Th IEEE international conference on advanced video and signal based surveillance (AVSS). IEEE, pp 343–348

  50. Nayak NM, Kamal AT, Roy-Chowdhury AK (2011) Vector field analysis for motion pattern identification in video. In: 2011 18Th IEEE international conference on image processing. IEEE, pp 2089–2092

  51. Nayak NM, Zhu Y, Roy-Chowdhury AK (2013) Vector field analysis for multi-object behavior modeling. Image Vis Comput 31(6-7):460–472

    Article  Google Scholar 

  52. Nguyen TN, Meunier J (2019) Anomaly detection in video sequence with appearance-motion correspondence

  53. Nievas EB, Suarez OD, Garcia GB, Sukthankar R (2011) Hockey fight detection dataset. In: Computer Analysis of Images and Patterns. Springer, pp 332–339. http://visilab.etsii.uclm.es/personas/oscar/FightDetection/

  54. Ohmura J, Egashira A, Satoh S, Miyoshi T, Irie H, Yoshinaga T (2011) Multi-gpu acceleration of optical flow computation in visual functional simulation. In: 2011 Second international conference on networking and computing. IEEE, pp 228– 234

  55. Pang G, Yan C, Shen C, Hengel Avd, Bai X (2020) Self-trained deep ordinal regression for end-to-end video anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

  56. Park H, Noh J, Ham B (2020) Learning memory-guided normality for anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

  57. Radford A, Metz L, Chintala S (2016) Unsupervised representation learning with deep convolutional generative adversarial networks. 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings. arXiv:1511.06434

  58. Rejitha MR, George SN (2019) An Unsupervised Abnormal Crowd Behavior Detection Technique using Farneback Algorithm. In: 2019 IEEE International conference on electronics, computing and communication technologies, CONECCT 2019, pp 1–5. https://doi.org/10.1109/CONECCT47791.2019.9012845

  59. Ruiz A, Lopez-de Teruel PE (2009) Diagram techniques for multiple view geometry. Proceedings of the IEEE International Conference on Computer Vision, pp 1865–1872. https://doi.org/10.1109/ICCV.2009.5459414

  60. Schlegl T, Seeböck P, Waldstein SM, Schmidt-Erfurth U, Langs G (2017) Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol 10265 LNCS, pp 146–147. https://doi.org/10.1007/978-3-319-59050-9_12

  61. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. arXiv:1406.2199

  62. Singh K, Yamini Preethi K, Vineeth Sai K, Modi CN (2018) Designing an Efficient Framework for Violence Detection in Sensitive Areas using Computer Vision and Machine Learning Techniques. In: 2018 10Th international conference on advanced computing, ICoAC 2018, pp 74–79. https://doi.org/10.1109/ICoAC44903.2018.8939110

  63. Stephens K (2016) Human and group activity recognition from video sequences. Ph.D. thesis, University of York

  64. Stephens K, Bors AG (2016) Grouping multi-vector streaklines for human activity identification. In: 2016 IEEE 12Th image, video, and multidimensional signal processing workshop, IVMSP 2016, pp 1–5. https://doi.org/10.1109/IVMSPW.2016.7528185

  65. Sudhakaran S, Lanz O (2017) Learning to detect violent videos using convolutional long short-term memory. 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance, AVSS 2017 pp 1–6. https://doi.org/10.1109/AVSS.2017.8078468. http://www.researchgate.net/publication/320662592

  66. Sun Q, Liu H, Harada T (2016) Online growing neural gas for anomaly detection in changing surveillance scenes. Pattern Recogn:S0031320316302771

  67. Van Wijk JJ (2002) Image based flow visualization. In: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’02, pp 745–754. https://doi.org/10.1145/566570.566646

  68. Wang L, Xiong Y, Wang Z, Qiao Y, Lin D, Tang X, van Gool L (2016) Temporal segment networks: Towards good practices for deep action recognition. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol 9912 LNCS, pp 20–36. https://doi.org/10.1007/978-3-319-46484-8_2

  69. Wang P, Ji Q (2007) Multi-view face and eye detection using discriminant features. Comput Vis Image Underst 105(2):99–111

    Article  Google Scholar 

  70. Wang X, Ma X, Grimson WEL (2009) Unsupervised activity perception in crowded and complicated scenes using hierarchical bayesian models. IEEE Trans Pattern Anal Mach Intell 31(3):539–555. https://doi.org/10.1109/TPAMI.2008.87

    Article  Google Scholar 

  71. Wang X, Qi C (2016) Action recognition using edge trajectories and motion acceleration descriptor. Mach Vis Appl 27(6):861–875

    Article  Google Scholar 

  72. Wang Y, Zhang Q, Li B (2016) Efficient unsupervised abnormal crowd activity detection based on a spatiotemporal saliency detector. 2016 IEEE Winter Conference on Applications of Computer Vision, WACV 2016 pp 1–9. https://doi.org/10.1109/WACV.2016.7477684. http://www.researchgate.net/publication/303563879

  73. Wu P, Liu J (2021) Learning causal temporal relation and feature discrimination for anomaly detection. IEEE Trans Image Process 30:3513–3527. https://doi.org/10.1109/TIP.2021.3062192

    Article  Google Scholar 

  74. Wu S, Wong HS (2012) Joint segmentation of collectively moving objects using a bag-of-words model and level set evolution. Pattern Recogn 45(9):3389–3401

    Article  Google Scholar 

  75. Xiong G, Cheng J, Wu X, Chen YL, Ou Y, Xu Y (2012) An energy model approach to people counting for abnormal crowd behavior detection. Neurocomputing 83:121–135. https://doi.org/10.1016/j.neucom.2011.12.007

    Article  Google Scholar 

  76. Xu D, Ricci E, Yan Y, Song J, Sebe N (2015) Learning Deep Representations of Appearance and Motion for Anomalous Event Detection, pp 8.1-8.12. https://doi.org/10.5244/c.29.8

  77. Xu D, Ricci E, Yan Y, Song J, Sebe N (2015) Learning deep representations of appearance and motion for anomalous event detection. Computer Vision and Image Understanding

  78. Xu L, Gong C, Yang J, Wu Q, Yao L (2014) Violent video detection based on moSIFT feature and sparse coding. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, pp 3538–3542. https://doi.org/10.1109/ICASSP.2014.6854259

  79. Yan S, Smith JS, Lu W, Zhang B (2020) Abnormal event detection from videos using a two-stream recurrent variational autoencoder. IEEE Trans Cogni Dev Syst 12(1):30–42. https://doi.org/10.1109/TCDS.2018.2883368

    Article  Google Scholar 

  80. Zenati H, Foo CS, Lecouat B, Manek G, Chandrasekhar VR (2018) Efficient gan-based anomaly detection. arXiv:1802.06222

  81. Zhang T, Jia W, Gong C, Sun J, Song X (2018) Semi-supervised dictionary learning via local sparse constraints for violence detection. Pattern Recogn Lett 107:98–104

    Article  Google Scholar 

  82. Zhang T, Jia W, Yang B, Yang J, He X, Zheng Z (2017) MoWLD: a robust motion image descriptor for violence detection. Multimed Tools Appl 76(1):1419–1438. https://doi.org/10.1007/s11042-015-3133-0

    Article  Google Scholar 

  83. Zhang T, Yang Z, Jia W, Yang B, Yang J, He X (2016) A new method for violence detection in surveillance scenes. Multimed Tools Appl 75(12):7327–7349. https://doi.org/10.1007/s11042-015-2648-8

    Article  Google Scholar 

  84. Zhao X, Gong D, Medioni G (2012) Tracking using motion patterns for very crowded scenes. In: European conference on computer vision. Springer, pp 315–328

  85. Zhao X, Medioni G (2011) Robust unsupervised motion pattern inference from video and applications. In: International conference on computer vision

  86. Zhou JT, Du J, Zhu H, Peng X, Liu Y, Goh RSM (2019) Anomalynet: an anomaly detection network for video surveillance. IEEE Trans Inf Forensic Secur 14(10):2537–2550. https://doi.org/10.1109/TIFS.2019.2900907

    Article  Google Scholar 

  87. Zhou JT, Du J, Zhu H, Peng X, Liu Y, Goh RSM (2019) Anomalynet: an anomaly detection network for video surveillance. IEEE Trans Inf Forensic Secur 14(10):2537–2550

    Article  Google Scholar 

  88. Zhou P, Ding Q, Luo H, Hou X (2018) Violence detection in surveillance video using low-level features. PLos One 13(10):e0203668. https://doi.org/10.1371/journal.pone.0203668

Download references

Acknowledgements

This work is partially supported by the National Natural Science Foundation of China (Nos.61601266, 61801272 and 61861038 ).

Author information

Authors and Affiliations

  1. School of Computer Science, Southwest Petroleum University, Chengdu, 610550, People’s Republic of China

    Jun Jiang, XinYue Wang, Chengyuan Zhao & Jia Wang

  2. School of Electrical and Electronic Engineering, Shandong University of Technology, Zibo, 255000, People’s Republic of China

    Mingliang Gao & Jinfeng Pan

Authors
  1. Jun Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. XinYue Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingliang Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jinfeng Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chengyuan Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jia Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingliang Gao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, J., Wang, X., Gao, M. et al. Abnormal behavior detection using streak flow acceleration. Appl Intell 52, 10632–10649 (2022). https://doi.org/10.1007/s10489-021-02881-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02881-7

Keywords

Search

Navigation