Skip to main content
SpringerLink
Log in

Abnormal activity detection using shear transformed spatio-temporal regions at the surveillance network edge

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper presents a method of detecting abnormal activity in crowd videos while considering the direction of the dominant crowd motion. One main goal of our approach is to be able to run at the edge of the surveillance network close to the surveillance cameras so as to reduce network congestion and decision latency. To capture motion features while considering the direction of dominant crowd direction we propose a generalised shear transform based spatio-temporal region. To detect abnormal activity, an autoencoder based method is adopted considering the requirement for running the method at the network edge. During training, the autoencoder learns motion features for each spatio-temporal region from video frames containing normal activity. While testing, those motion features from each spatio-temporal region that cannot be reconstructed satisfactorily by the autoencoder indicate abnormal activity. This approach allows coarse localisation as well as detection of abnormal activity. The approach demonstrated \(\mathcal {O}(n)\) behaviour with ability to work at higher frame rates by trading off accuracy. The approach has been verified against recent works on standard abnormal activity datasets: UCSD dataset and Subway dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Adam A, Rivlin E, Shimshoni I, Reinitz D (2008) Robust real-time unusual event detection using multiple fixed-location monitors. IEEE Trans Pattern Anal Mach Intell 30(3):555–560

    Article  Google Scholar 

  2. Afiq AA, et al. (2019) A review on classifying abnormal behavior in crowd scene. J Vis Commun Image Represent 58:285–303

    Article  Google Scholar 

  3. Biswas S, Babu RV (2017) Anomaly detection via short local trajectories. Neurocomputing 242: 63–72

    Article  Google Scholar 

  4. Bouguet JY (2000) Pyramidal implementation of the Lucas Kanade feature tracker description of the algorithm. OpenCV Document, Intel, Microprocessor Research Labs

  5. Bouindour S, Hu R, Snoussi H (2019) Enhanced convolutional neural network for abnormal event detection in video streams. IEEE Int Conf Artif Intell Knowl Eng (AIKE) 172–178

  6. Chalapathy R, Chawla S (2019) Deep learning for anomaly detection: a survey. arXiv:1901.03407

  7. Chaudhry R, Ravichandran A, Hager G, Vidal R (2009) Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions. IEEE Conf Comput Vis Pattern Recognit 1932–1939

  8. Chen N, Chen Y, Blasch E, Ling H, You Y, Ye X (2017) Enabling smart urban surveillance at the edge. IEEE Int Conf Smart Cloud 109–119. https://doi.org/10.1109/SmartCloud.2017.24

  9. Chong YS, Tay YH (2017) Abnormal event detection in videos using spatiotemporal Autoencoder. Int Symp Neural Netw. 189–196

  10. Cicirelli F, et al. (2018) Edge computing and social internet of things for Large-Scale smart environments development. IEEE Internet Things J 5(4):2557–2571. https://doi.org/10.1109/JIOT.2017.2775739

    Article  Google Scholar 

  11. Cisco Annual Internet Report, 2018–2023 - Whitepaper (2020)

  12. Cisco Visual Networking Index: Complete Forecast Update, 2017–2022 - White Paper (2018)

  13. Colque RVHM, Junior CAC, Schwartz WR (2015) Histograms of optical flow orientation and magnitude to detect anomalous events in videos. SIBGRAPI Conf Graph Patterns Images 126–133

  14. Colque RVHM, Caetano C, de Andrade MTL, Schwartz WR (2017) Histograms of optical flow orientation and magnitude and entropy to detect anomalous events in videos. IEEE Trans Circ Sys Video Tech 27(3):673–682

    Article  Google Scholar 

  15. Colque RM, et al. (2018) Novel anomalous event detection based on human-object interactions. Int Conf Comput Vis Theory Appl 293–300

  16. Fan Y, Wen G, Li D, Qiu S, Levine MD (2018) Video anomaly detection and localization via gaussian mixture fully convolutional variational autoencoder. arXiv:1805.11223

  17. George M, Jose BR, Mathew J, Kokare P (2019) Autoencoder-based abnormal activity detection using parallelepiped spatiotemporal region. IET Comput Vis 13(1):23–30. https://doi.org/10.1049/iet-cvi.2018.5240

    Article  Google Scholar 

  18. Hasan M, Choi J, Neumann J, Roy-Chowdhury AK, Davis L (2016) Learning temporal regularity in video sequences. IEEE Conf Comput Vis Pattern Recognit 733–742. https://doi.org/10.1109/CVPR.2016.86

  19. He Y, Zhao J (2019) Temporal convolutional networks for anomaly detection in time series. J Phys Conf Ser 1213

  20. Howard AG, et al. (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861

  21. Hu X, Huang Y, Gao X, Luo L, Duan Q (2019) Squirrel-cage local binary pattern and its application in video anomaly detection. IEEE Trans Inf Forensics Secur 14(4):1007–1022

    Article  Google Scholar 

  22. Khan MUK, Park H, Kyung C (2019) Rejecting motion outliers for efficient crowd anomaly detection. IEEE Trans Inf Forensics Secur 14(2):541–556. https://doi.org/10.1109/TIFS.2018.2856189

    Article  Google Scholar 

  23. Klaser A, Marszalek M, Schmid C (2008) A spatio-temporal descriptor based on 3D-gradients. Br Mach Vis Conf 275:1–10

    Google Scholar 

  24. Lea C, Flynn M, Vidal R, Reiter A, Hager G (2017) Temporal convolutional networks for action segmentation and detection. IEEE Conf Comput Vis Pattern Recognit (CVPR) 1003–1012. https://doi.org/10.1109/CVPR.2017.113

  25. Lei Z, Deng F, Yang X (2019) Spatial temporal balanced generative adversarial autoencoder for anomaly detection. Int Conf Image Video Signal Process 1–7

  26. Leyva R, Sanchez V, Li CT (2017) Video anomaly detection with compact feature sets for online performance. IEEE Trans Image Proc 26(7):3463–3478

    Article  MathSciNet  Google Scholar 

  27. Li W, Mahadevan V, Vasconcelos N (2014) Anomaly detection and localization in crowded scenes. IEEE Trans Pattern Anal Mach Intell 36(1):18–32

    Article  Google Scholar 

  28. Liu P, Tao Y, Zhao W, Tang X (2017) Abnormal crowd motion detection using double sparse representation. Neurocomputing 269:3–12

    Article  Google Scholar 

  29. Lloyd K, Marshall D, Moore SC, Rosin PL (2017) Detecting violent and abnormal crowd activity using temporal analysis of grey level co-occurrence matrix (GLCM) based texture measures. Mach Vis Appl 28:361–371

    Article  Google Scholar 

  30. Lu C, Shi J, Wang W, Jia J (2018) Fast abnormal event detection. Int J Comput Vis 1–18

  31. Mabrouk AB, Zagrouba E (2018) Abnormal behavior recognition for intelligent video surveillance systems: a review. Expert Syst Appl 91:480–491

    Article  Google Scholar 

  32. Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. Proc IEEE Conf Comput Vis Pattern Recognit (CVPR) 1975–1981

  33. Miraftabzadeh SA, Rad P, Choo KR, Jamshidi M (2018) A privacy-aware architecture at the edge for autonomous real-time identity reidentification in crowds. IEEE Internet Things J 5(4):2936–2946. https://doi.org/10.1109/JIOT.2017.2761801

    Article  Google Scholar 

  34. Nikouei SY, Chen Y, Song S, Xu R, Choi BY, Faughnan T (2018) Real-time human detection as an edge service enabled by a lightweight CNN. IEEE Int Conf Edge Comput 125–129

  35. Nikouei SY, Chen Y, Song S, Xu R, Choi BY, Faughnan T (2018) Intelligent surveillance as an edge network service: from harr-cascade, SVM to a lightweight CNN. arXiv:1805.00331

  36. Quigley PA, et al. (2019) Outcomes of patient-engaged video surveillance on falls and other adverse events. Clin Geriatr Med 35(2):253–263

    Article  MathSciNet  Google Scholar 

  37. Rabiee H, Mousavi H, Nabi M, Ravanbakhsh M (2017) Detection and localization of crowd behavior using a novel tracklet-based model. Int J Mach Learn Cybern 1–12

  38. Ravanbakhsh M, Nabi M, Mousavi H, Sangineto E, Sebe N (2018) Plug-and-play CNN for crowd motion analysis: an application in abnormal event detection. IEEE Winter Conf Appl Comput Vis (WACV) 1689–1698

  39. Rumelhart D, Hinton G, Williams R (1986) Learning representations by back-propagating errors. Nature 323:533–536. https://doi.org/10.1038/323533a0

    Article  MATH  Google Scholar 

  40. Shi W, Cao J, Zhang Q, Li Y, Xu L (2016) Edge computing: vision and challenges. IEEE Internet Things J 3(5):637–646

    Article  Google Scholar 

  41. Shi X, Chen Z, Wang H, Yeung DY, Wong W, Woo W (2015) Convolutional LSTM network: a machine learning approach for precipitation nowcasting. Int Conf Neural Inf Process Syst 802–810

  42. Sun J, Wang X, Xiong N, Shao J (2018) Learning sparse representation with variational Auto-Encoder for anomaly detection. IEEE Access 6:33353–33361. https://doi.org/10.1109/ACCESS.2018.2848210

    Article  Google Scholar 

  43. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. Int Conf Neural Inf Process Syst 3104–3112

  44. Szegedy C, et al. (2015) Going deeper with convolutions. IEEE Conf Comput Vis Pattern Recognit (CVPR) 1–9

  45. Tsakanikas V, Dagiuklas T (2017) Video surveillance systems-current status and future trends. Comput Electr Eng. https://doi.org/10.1016/j.compeleceng.2017.11.011

  46. Usman M, Jan MA, He X, Chen J (2019) A survey on big multimedia data processing and management in smart cities. ACM Comput Surv 52 (3):54:1–54:29

    Article  Google Scholar 

  47. Wang J, Xu Z (2016) Spatio-temporal texture modelling for real-time crowd anomaly detection. Comput Vis Image Underst 144:177–187

    Article  Google Scholar 

  48. Wang J, Cherian A, Porikli F (2017) Ordered pooling of optical flow sequences for action recognition. IEEE Winter Conf Appl Comput Vis (WACV) 168–176. https://doi.org/10.1109/WACV.2017.26

  49. Wang L, Zhou F, Li Z, Zuo W, Tan H (2018) Abnormal event detection in videos using hybrid spatio-temporal autoencoder. IEEE Int Conf Image Process 2276–2280

  50. Werbos PJ (1990) Backpropagation through time: what it does and how to do it. Proc IEEE 78(10):1550–1560

    Article  Google Scholar 

  51. Xu R, et al. (2018) Real-time human objects tracking for smart surveillance at the edge. IEEE Int Conf Commun 1–6. https://doi.org/10.1109/ICC.2018.8422970

  52. Yang B, Cao J, Ni R, Zou L (2018) Anomaly detection in moving crowds through spatiotemporal autoencoding and additional attention. Adv Multimed 2018:1–8

    Google Scholar 

  53. Yuan Y, Feng Y, Lu X (2017) Statistical hypothesis detector for abnormal event detection in crowded scenes. IEEE Trans Cybern 47(11):3597–3608

    Article  Google Scholar 

  54. Yuan Y, Feng Y, Lu X (2018) Structured dictionary learning for abnormal event detection in crowded scenes. Pattern Recognit 73:99–110

    Article  Google Scholar 

  55. Zhang T, et al. (2015) The design and implementation of a wireless video surveillance system. Annu Int Conf Mob Comput Netw 426–438

  56. Zhou F, Wang L, Li Z, Zuo W, Tan H (2019) Unsupervised learning approach for abnormal event detection in surveillance video by hybrid autoencoder. Neural Process Lett. https://doi.org/10.1007/s11063-019-10113-w

  57. Zitouni MS, Sluzek A, Bhaskar H (2019) Visual analysis of socio-cognitive crowd behaviors for surveillance: a survey and categorization of trends and methods. Eng Appl Artif Intell 82:294–312

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by Technical Education Quality Improvement Programme (TEQIP) Research Seed Money Project (No. TEQIP/PTRA/2017); APJ Abdul Kalam Technological University - Center for Engineering Research & Development (APJAKTU-CERD) Research Seed Money Project (No. KTU/RESEARCH 2/4068/2019).

Author information

Authors and Affiliations

  1. School of Engineering, Cochin University of Science and Technology, Kochi, Kerala, 682022, India

    Michael George & Babita Roslind Jose

  2. Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, Bihar, 801103, India

    Jimson Mathew

Authors
  1. Michael George
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Babita Roslind Jose
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jimson Mathew
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael George.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

George, M., Jose, B.R. & Mathew, J. Abnormal activity detection using shear transformed spatio-temporal regions at the surveillance network edge. Multimed Tools Appl 79, 27511–27532 (2020). https://doi.org/10.1007/s11042-020-09277-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09277-8

Keywords

Search

Navigation