Skip to main content
SpringerLink
Log in

A new method of abnormal behavior detection using LSTM network with temporal attention mechanism

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

In this paper, we propose an accurate and effective method for detecting abnormal behavior. We consider the video as a series of frame sequences; in the training phase, our deep learning framework is used to extract appearance features and learn the relationship between historical features and current features in the normal video. In the testing phase, the predicted features that differ from the actual features are considered as abnormal. Our model is designed as a feature prediction framework with a new temporal attention mechanism. In the feature extraction stage, we transform a pre-trained Vgg16 network into a fully convolutional neural network and used the third pooling layer output as the appearance feature extraction to effectively capture static appearance features. Then, a new temporal attention mechanism is introduced to learn the contribution of different historical appearance features at the same position to the current features, thereby solving the problem of representing dynamic motion features. Finally, the LSTM network is used to decode the historical feature sequences with temporal attention to predict the features at the current moment. Those actual features that differ from the predicted features are considered as abnormal features. Using upsampling for the abnormal features locates abnormal behavior on the original frames. Experiments on two benchmark datasets demonstrate the competitive performance of our method with the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Mingliang X, Li C, Lv P, Lin N, Hou R, Zhou B (2017) An efficient method of crowd aggregation computation in public areas. IEEE Trans Circuits Syst Video Technol 28(10):2814–2825

    Google Scholar 

  2. Xie S, Zhang X, Cai J (2019) Video crowd detection and abnormal behavior model detection based on machine learning method. Neural Comput Appl 31:175–184

    Article  Google Scholar 

  3. Shen M, Jiang X, Sun T (2018) Anomaly detection based on nearest neighbor search with locality-sensitive B-tree. Neurocomputing 289:55–67

    Article  Google Scholar 

  4. Xing H, Huang Y, Duan Q, Ci W, Dai J, Yang H (2018) Abnormal event detection in crowded scenes using histogram of oriented contextual gradient descriptor. EURASIP J Adv Signal Process 2018(1):54

    Article  Google Scholar 

  5. Ke X, Jiang X, Sun T (2018) Anomaly detection based on stacked sparse coding with intraframe classification strategy. IEEE Trans Multimed 20(5):1062–1074

    Article  Google Scholar 

  6. Sabokrou M, Fayyaz M, Fathy M, Klette R (2017) Deep-cascade: cascading 3D deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans Image Process 26(4):1992–2004

    Article  MathSciNet  Google Scholar 

  7. Coşar S, Donatiello G, Bogorny V, Garate C, Alvares LO, Brémond F (2017) Toward abnormal trajectory and event detection in video surveillance. IEEE Trans Circuits Syst Video Technol 27(3):683–695

    Article  Google Scholar 

  8. Ye O, Deng J, Yu Z, Liu T, Dong L (2020) Abnormal event detection via feature expectation subgraph calibrating classification in video surveillance scenes. IEEE Access 8:97564–97575

    Article  Google Scholar 

  9. Bosi Y, Liu Y, Sun Q (2016) A content-adaptively sparse reconstruction method for abnormal events detection with low-rank property. IEEE Trans Syst Man Cybern Syst 47(4):704–716

    Google Scholar 

  10. Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp 935–942

  11. Fernando T, Denman S, Sridharan S, Fookes C (2018) Soft+ hardwired attention: an lstm framework for human trajectory prediction and abnormal event detection. Neural Netw 108:466–478

    Article  Google Scholar 

  12. Ullah A, Muhammad K, Del Ser J, Baik SW, de Albuquerque VHC (2018) Activity recognition using temporal optical flow convolutional features and multilayer lstm. IEEE Trans Ind Electron 66(12):9692–9702

    Article  Google Scholar 

  13. Martinel N, Micheloni C, Piciarelli C, Foresti GL (2013) Camera selection for adaptive human–computer interface. IEEE Trans Syst Man Cybern Syst 44(5):653–664

    Article  Google Scholar 

  14. Sabokrou M, Fathy M, Hoseini M, Klette R (2015) Real-time anomaly detection and localization in crowded scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 56–62

  15. Liu L, Wang S, Hu B, Qiong Q, Wen J, Rosenblum DS (2018) Learning structures of interval-based bayesian networks in probabilistic generative model for human complex activity recognition. Pattern Recognit 81:545–561

    Article  Google Scholar 

  16. Epaillard E, Bouguila N (2018) Variational Bayesian learning of generalized dirichlet-based hidden Markov models applied to unusual events detection. IEEE Trans Neural Netw Learn Syst 30(4):1034–1047

    Article  MathSciNet  Google Scholar 

  17. Ouivirach K, Gharti S, Dailey MN (2013) Incremental behavior modeling and suspicious activity detection. Pattern Recognit 46(3):671–680

    Article  Google Scholar 

  18. Kratz L, Nishino K (2009) Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp 1446–1453

  19. Li J, Gong S, Xiang T (2008) Global behaviour inference using probabilistic latent semantic analysis. In: BMVC, vol 3231, p 3232

  20. Isupova O, Kuzin D, Mihaylova L (2018) Learning methods for dynamic topic modeling in automated behavior analysis. IEEE Trans Neural Netw Learn Syst 29(9):3980–3993

    Article  Google Scholar 

  21. Hospedales T, Gong S, Xiang T (2012) Video behaviour mining using a dynamic topic model. Int J Comput Vis 98(3):303–323

    Article  MathSciNet  Google Scholar 

  22. Tang X, Zhang S, Yao H (2013) Sparse coding based motion attention for abnormal event detection. In: 2013 IEEE International Conference on Image Processing. IEEE, pp 3602–3606

  23. Ren H, Moeslund TB (2014) Abnormal event detection using local sparse representation. In: 2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). IEEE, pp 125–130

  24. He C, Shao J, Sun J (2018) An anomaly-introduced learning method for abnormal event detection. Multimed Tools Appl 77(22):29573–29588

    Article  Google Scholar 

  25. Sun J, Wang X, Xiong N, Shao J (2018) Learning sparse representation with variational auto-encoder for anomaly detection. IEEE Access 6:33353–33361

    Article  Google Scholar 

  26. Ko K-E, Sim K-B (2018) Deep convolutional framework for abnormal behavior detection in a smart surveillance system. Eng Appl Artif Intell 67:226–234

    Article  Google Scholar 

  27. Sabokrou M, Fayyaz M, Fathy M, Moayed Z, Klette R (2018) Deep-anomaly: fully convolutional neural network for fast anomaly detection in crowded scenes. Comput Vis Image Underst 172:88–97

    Article  Google Scholar 

  28. Chu W, Xue H, Yao C, Cai D (2018) Sparse coding guided spatiotemporal feature learning for abnormal event detection in large videos. IEEE Trans Multimed 21(1):246–255

    Article  Google Scholar 

  29. Fang Z, Fei F, Fang Y, Lee C, Xiong N, Shu L, Chen S (2016) Abnormal event detection in crowded scenes based on deep learning. Multimed Tools Appl 75(22):14617–14639

    Article  Google Scholar 

  30. Zhou JT, Du J, Zhu H, Peng X, Liu Y, Goh RSM (2019) Anomalynet: an anomaly detection network for video surveillance. IEEE Trans Inf Forensics Secur 14(10):2537–2550

    Article  Google Scholar 

  31. Feng Y, Yuan Y, Xiaoqiang L (2017) Learning deep event models for crowd anomaly detection. Neurocomputing 219:548–556

    Article  Google Scholar 

  32. Chan T-H, Jia K, Gao S, Jiwen L, Zeng Z, Ma Y (2015) PCANet: a simple deep learning baseline for image classification? IEEE Trans Image Process 24(12):5017–5032

    Article  MathSciNet  Google Scholar 

  33. Yong B, Zhang G, Chen H, Zhou Q (2017) Intelligent monitor system based on cloud and convolutional neural networks. J Supercomput 73(7):3260–3276

    Article  Google Scholar 

  34. Zhao L, Su C, Dai Z, Huang H, Ding S, Huang X, Han Z (2019) Indoor device-free passive localization with DCNN for location-based services. J Supercomput. https://doi.org/10.1007/s11227-019-03110-2

    Article  Google Scholar 

  35. Xiao Y, Yu D, Wang X, Lv Y, Fan T, Wu L (2020) SPCNet: spatial preserve and content-aware network for human pose estimation. arXiv e-prints arXiv:2004.05834

  36. Tay N, Connie T, Ong TS, Teh PS (2019) Abnormal behavior recognition using CNN-LSTM with attention mechanism. In: 2019 1st International Conference on Electrical, Control and Instrumentation Engineering (ICECIE), pp 1–5

  37. Liu W, Luo W, Lian D, Gao S (2018) Future frame prediction for anomaly detection—a new baseline. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6536–6545

  38. Woo S, Park J, Lee J-Y, So Kweon I (2018) Cbam: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 3–19

  39. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2921–2929

  40. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7132–7141

  41. Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, pp 1975–1981

  42. Lu C, Shi J, Jia J (2013) Abnormal event detection at 150 fps in matlab. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2720–2727

Download references

Author information

Authors and Affiliations

  1. School of Automation, Central South University, Changsha, China

    Limin Xia & Zhenmin Li

Authors
  1. Limin Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenmin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhenmin Li.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xia, L., Li, Z. A new method of abnormal behavior detection using LSTM network with temporal attention mechanism. J Supercomput 77, 3223–3241 (2021). https://doi.org/10.1007/s11227-020-03391-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-020-03391-y

Keywords

Search

Navigation