skip to main content
10.1145/3465481.3470099acmotherconferencesArticle/Chapter ViewAbstractPublication PagesaresConference Proceedingsconference-collections
research-article

A Real-Time Deep Learning Approach for Real-World Video Anomaly Detection

Published: 17 August 2021 Publication History

Abstract

Anomaly detection in video streams with imbalanced data and real-time constraints is a challenging task of computer vision. This paper proposes a novel real-time approach for real-world video anomaly detection exploiting a supervised learning methodology. In particular, we present a deep learning architecture based on the analysis of contextual, spatial, and motion information extracted from the video. A data balancing strategy based on hard-mining and adaptive framerate is used to avoid overfitting and increase detection accuracy. The approach defines an extended taxonomy by differentiating anomalies in ”soft” and ”hard”. A novel anomaly detection score based on a sigmoidal function has been introduced to reduce false positive rate while maintaining a high level of true positive rate. The proposed methodology has been validated with a set of experiments on a well-known video anomaly dataset: UCF-CRIME. The experiments on the testbed demonstrate the impact of the contextual information and data balancing on the classification performances, considering only ”hard” anomalies during training and that the proposed model can achieve state-of-the-art performances while minimizing resource consumption.

References

[1]
2020. A Survey on Deep Learning Techniques for Video Anomaly Detection. arXiv preprint arXiv: 2009.14146(2020).
[2]
Ejaz Ahmed, Michael Jones, and Tim K Marks. 2015. An improved deep learning architecture for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3908–3916.
[3]
Sukalyan Bhakat and Ganesh Ramakrishnan. 2019. Anomaly Detection in Surveillance Videos. In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data (Kolkata, India) (CoDS-COMAD ’19). Association for Computing Machinery, New York, NY, USA, 252–255. https://doi.org/10.1145/3297001.3297034
[4]
Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. 2020. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934(2020).
[5]
Lokesh Boominathan, Srinivas SS Kruthiventi, and R Venkatesh Babu. 2016. Crowdnet: A deep convolutional network for dense crowd counting. In Proceedings of the 24th ACM international conference on Multimedia. 640–644.
[6]
Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6299–6308.
[7]
Rensso Victor Hugo Mora Colque, Carlos Caetano, Matheus Toledo Lustosa de Andrade, and William Robson Schwartz. 2016. Histograms of optical flow orientation and magnitude and entropy to detect anomalous events in videos. IEEE Transactions on Circuits and Systems for Video Technology 27, 3(2016), 673–682.
[8]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.
[9]
Qi Dong, Shaogang Gong, and Xiatian Zhu. 2017. Class rectification hard mining for imbalanced deep learning. In Proceedings of the IEEE International Conference on Computer Vision. 1851–1860.
[10]
Keval Doshi and Yasin Yilmaz. 2021. Online anomaly detection in surveillance videos with asymptotic bound on false alarm rate. Pattern Recognition 114(2021), 107865.
[11]
Giacomo Giorgi, Antonio La Marra, Fabio Martinelli, Paolo Mori, and Andrea Saracino. 2017. Smart parental advisory: A usage control and deep learning-based framework for dynamic parental control on smart TV. In International Workshop on Security and Trust Management. Springer, 118–133.
[12]
Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. 2019. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1705–1714.
[13]
Mahmudul Hasan, Jonghyun Choi, Jan Neumann, Amit K Roy-Chowdhury, and Larry S Davis. 2016. Learning temporal regularity in video sequences. In Proceedings of the IEEE conference on computer vision and pattern recognition. 733–742.
[14]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
[15]
Jordan Henrio and Tomoharu Nakashima. 2018. Anomaly Detection in Videos Recorded by Drones in a Surveillance Context. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2503–2508.
[16]
Berthold KP Horn and Brian G Schunck. 1981. Determining optical flow. Artificial intelligence 17, 1-3 (1981), 185–203.
[17]
Earnest Paul Ijjina and Krishna Mohan Chalavadi. 2017. Human action recognition in RGB-D videos using motion sequence information and deep learning. Pattern Recognition 72(2017), 504–516.
[18]
Mehrsan Javan Roshtkhari and Martin D Levine. 2013. Online dominant and anomalous behavior detection in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2611–2618.
[19]
Fan Jiang, Junsong Yuan, Sotirios A Tsaftaris, and Aggelos K Katsaggelos. 2011. Anomalous video event detection using spatiotemporal context. Computer Vision and Image Understanding 115, 3 (2011), 323–333.
[20]
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 1725–1732.
[21]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012), 1097–1105.
[22]
Federico Landi, Cees GM Snoek, and Rita Cucchiara. 2019. Anomaly locality in video surveillance. arXiv preprint arXiv:1901.10364(2019).
[23]
Ce Li, Zhenjun Han, Qixiang Ye, and Jianbin Jiao. 2013. Visual abnormal behavior detection based on trajectory sparse reconstruction analysis. Neurocomputing 119(2013), 94–100.
[24]
Weixin Li, Vijay Mahadevan, and Nuno Vasconcelos. 2013. Anomaly detection and localization in crowded scenes. IEEE transactions on pattern analysis and machine intelligence 36, 1(2013), 18–32.
[25]
Yuanyuan Li, Yiheng Cai, Jiaqi Liu, Shinan Lang, and Xinfeng Zhang. 2019. Spatio-temporal unity networking for video anomaly detection. IEEE Access 7(2019), 172425–172432.
[26]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740–755.
[27]
Hong Liu, Juanhui Tu, and Mengyuan Liu. 2017. Two-stream 3d convolutional neural network for skeleton-based action recognition. arXiv preprint arXiv:1705.08106(2017).
[28]
Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. 2018. Future frame prediction for anomaly detection–a new baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6536–6545.
[29]
Vijay Mahadevan, Wei-Xin LI, Viral Bhalodia, and Nuno Vasconcelos. 2010. Anomaly Detection in Crowded Scenes. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 1975–1981.
[30]
Ramin Mehran, Alexis Oyama, and Mubarak Shah. 2009. Abnormal crowd behavior detection using social force model. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 935–942.
[31]
Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784(2014).
[32]
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision. 618–626.
[33]
Karen Simonyan and Andrew Zisserman. 2014. Two-stream convolutional networks for action recognition in videos. arXiv preprint arXiv:1406.2199(2014).
[34]
Waqas Sultani, Chen Chen, and Mubarak Shah. 2018. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6479–6488.
[35]
Zheng Tang, Milind Naphade, Ming-Yu Liu, Xiaodong Yang, Stan Birchfield, Shuo Wang, Ratnesh Kumar, David Anastasiu, and Jenq-Neng Hwang. 2019. Cityflow: A city-scale benchmark for multi-target multi-camera vehicle tracking and re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8797–8806.
[36]
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision. 4489–4497.
[37]
Di Zang, Zhenliang Chai, Junqi Zhang, Dongdong Zhang, and Jiujun Cheng. 2015. Vehicle license plate recognition using visual attention model and deep learning. Journal of Electronic Imaging 24, 3 (2015), 033001.
[38]
Yi Zhu, Xinyu Li, Chunhui Liu, Mohammadreza Zolfaghari, Yuanjun Xiong, Chongruo Wu, Zhi Zhang, Joseph Tighe, R Manmatha, and Mu Li. 2020. A Comprehensive Study of Deep Video Action Recognition. arXiv preprint arXiv:2012.06567(2020).

Cited By

View all
  • (2023)Anomaly Detection in ATM Vestibules Using Three-Stream Deep Learning ApproachComputer Vision and Image Processing10.1007/978-3-031-31407-0_1(1-12)Online publication date: 7-May-2023
  • (2022)Evaluation of the effectiveness of a crowdsourcing-based crime detection systemIEICE Communications Express10.1587/comex.2022XBL009911:9(607-611)Online publication date: 1-Sep-2022
  • (2022)Anomaly detection in surveillance videos: a thematic taxonomy of deep models, review and performance analysisArtificial Intelligence Review10.1007/s10462-022-10258-656:4(3319-3368)Online publication date: 27-Aug-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ARES '21: Proceedings of the 16th International Conference on Availability, Reliability and Security
August 2021
1447 pages
ISBN:9781450390514
DOI:10.1145/3465481
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Anomaly detection
  2. Behavioral analysis
  3. Computer vision
  4. Deep Learning

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

ARES 2021

Acceptance Rates

Overall Acceptance Rate 228 of 451 submissions, 51%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)20
  • Downloads (Last 6 weeks)0
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Anomaly Detection in ATM Vestibules Using Three-Stream Deep Learning ApproachComputer Vision and Image Processing10.1007/978-3-031-31407-0_1(1-12)Online publication date: 7-May-2023
  • (2022)Evaluation of the effectiveness of a crowdsourcing-based crime detection systemIEICE Communications Express10.1587/comex.2022XBL009911:9(607-611)Online publication date: 1-Sep-2022
  • (2022)Anomaly detection in surveillance videos: a thematic taxonomy of deep models, review and performance analysisArtificial Intelligence Review10.1007/s10462-022-10258-656:4(3319-3368)Online publication date: 27-Aug-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media