Stefano Petrocchi
ABSTRACT
Anomaly detection in video streams with imbalanced data and real-time constraints is a challenging task of computer vision. This paper proposes a novel real-time approach for real-world video anomaly detection exploiting a supervised learning methodology. In particular, we present a deep learning architecture based on the analysis of contextual, spatial, and motion information extracted from the video. A data balancing strategy based on hard-mining and adaptive framerate is used to avoid overfitting and increase detection accuracy. The approach defines an extended taxonomy by differentiating anomalies in ”soft” and ”hard”. A novel anomaly detection score based on a sigmoidal function has been introduced to reduce false positive rate while maintaining a high level of true positive rate. The proposed methodology has been validated with a set of experiments on a well-known video anomaly dataset: UCF-CRIME. The experiments on the testbed demonstrate the impact of the contextual information and data balancing on the classification performances, considering only ”hard” anomalies during training and that the proposed model can achieve state-of-the-art performances while minimizing resource consumption.
- 2020. A Survey on Deep Learning Techniques for Video Anomaly Detection. arXiv preprint arXiv: 2009.14146(2020).Google Scholar
- Ejaz Ahmed, Michael Jones, and Tim K Marks. 2015. An improved deep learning architecture for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3908–3916.Google ScholarCross Ref
- Sukalyan Bhakat and Ganesh Ramakrishnan. 2019. Anomaly Detection in Surveillance Videos. In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data (Kolkata, India) (CoDS-COMAD ’19). Association for Computing Machinery, New York, NY, USA, 252–255. https://doi.org/10.1145/3297001.3297034Google ScholarDigital Library
- Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. 2020. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934(2020).Google Scholar
- Lokesh Boominathan, Srinivas SS Kruthiventi, and R Venkatesh Babu. 2016. Crowdnet: A deep convolutional network for dense crowd counting. In Proceedings of the 24th ACM international conference on Multimedia. 640–644.Google ScholarDigital Library
- Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6299–6308.Google ScholarCross Ref
- Rensso Victor Hugo Mora Colque, Carlos Caetano, Matheus Toledo Lustosa de Andrade, and William Robson Schwartz. 2016. Histograms of optical flow orientation and magnitude and entropy to detect anomalous events in videos. IEEE Transactions on Circuits and Systems for Video Technology 27, 3(2016), 673–682.Google ScholarDigital Library
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.Google ScholarCross Ref
- Qi Dong, Shaogang Gong, and Xiatian Zhu. 2017. Class rectification hard mining for imbalanced deep learning. In Proceedings of the IEEE International Conference on Computer Vision. 1851–1860.Google ScholarCross Ref
- Keval Doshi and Yasin Yilmaz. 2021. Online anomaly detection in surveillance videos with asymptotic bound on false alarm rate. Pattern Recognition 114(2021), 107865.Google ScholarCross Ref
- Giacomo Giorgi, Antonio La Marra, Fabio Martinelli, Paolo Mori, and Andrea Saracino. 2017. Smart parental advisory: A usage control and deep learning-based framework for dynamic parental control on smart TV. In International Workshop on Security and Trust Management. Springer, 118–133.Google ScholarCross Ref
- Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. 2019. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1705–1714.Google ScholarCross Ref
- Mahmudul Hasan, Jonghyun Choi, Jan Neumann, Amit K Roy-Chowdhury, and Larry S Davis. 2016. Learning temporal regularity in video sequences. In Proceedings of the IEEE conference on computer vision and pattern recognition. 733–742.Google ScholarCross Ref
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.Google ScholarCross Ref
- Jordan Henrio and Tomoharu Nakashima. 2018. Anomaly Detection in Videos Recorded by Drones in a Surveillance Context. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2503–2508.Google ScholarDigital Library
- Berthold KP Horn and Brian G Schunck. 1981. Determining optical flow. Artificial intelligence 17, 1-3 (1981), 185–203.Google Scholar
- Earnest Paul Ijjina and Krishna Mohan Chalavadi. 2017. Human action recognition in RGB-D videos using motion sequence information and deep learning. Pattern Recognition 72(2017), 504–516.Google ScholarDigital Library
- Mehrsan Javan Roshtkhari and Martin D Levine. 2013. Online dominant and anomalous behavior detection in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2611–2618.Google ScholarDigital Library
- Fan Jiang, Junsong Yuan, Sotirios A Tsaftaris, and Aggelos K Katsaggelos. 2011. Anomalous video event detection using spatiotemporal context. Computer Vision and Image Understanding 115, 3 (2011), 323–333.Google ScholarDigital Library
- Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 1725–1732.Google ScholarDigital Library
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012), 1097–1105.Google Scholar
- Federico Landi, Cees GM Snoek, and Rita Cucchiara. 2019. Anomaly locality in video surveillance. arXiv preprint arXiv:1901.10364(2019).Google Scholar
- Ce Li, Zhenjun Han, Qixiang Ye, and Jianbin Jiao. 2013. Visual abnormal behavior detection based on trajectory sparse reconstruction analysis. Neurocomputing 119(2013), 94–100.Google ScholarDigital Library
- Weixin Li, Vijay Mahadevan, and Nuno Vasconcelos. 2013. Anomaly detection and localization in crowded scenes. IEEE transactions on pattern analysis and machine intelligence 36, 1(2013), 18–32.Google Scholar
- Yuanyuan Li, Yiheng Cai, Jiaqi Liu, Shinan Lang, and Xinfeng Zhang. 2019. Spatio-temporal unity networking for video anomaly detection. IEEE Access 7(2019), 172425–172432.Google ScholarCross Ref
- Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740–755.Google ScholarCross Ref
- Hong Liu, Juanhui Tu, and Mengyuan Liu. 2017. Two-stream 3d convolutional neural network for skeleton-based action recognition. arXiv preprint arXiv:1705.08106(2017).Google Scholar
- Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. 2018. Future frame prediction for anomaly detection–a new baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6536–6545.Google ScholarCross Ref
- Vijay Mahadevan, Wei-Xin LI, Viral Bhalodia, and Nuno Vasconcelos. 2010. Anomaly Detection in Crowded Scenes. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 1975–1981.Google ScholarCross Ref
- Ramin Mehran, Alexis Oyama, and Mubarak Shah. 2009. Abnormal crowd behavior detection using social force model. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 935–942.Google ScholarCross Ref
- Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784(2014).Google Scholar
- Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision. 618–626.Google ScholarCross Ref
- Karen Simonyan and Andrew Zisserman. 2014. Two-stream convolutional networks for action recognition in videos. arXiv preprint arXiv:1406.2199(2014).Google Scholar
- Waqas Sultani, Chen Chen, and Mubarak Shah. 2018. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6479–6488.Google ScholarCross Ref
- Zheng Tang, Milind Naphade, Ming-Yu Liu, Xiaodong Yang, Stan Birchfield, Shuo Wang, Ratnesh Kumar, David Anastasiu, and Jenq-Neng Hwang. 2019. Cityflow: A city-scale benchmark for multi-target multi-camera vehicle tracking and re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8797–8806.Google ScholarCross Ref
- Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision. 4489–4497.Google ScholarDigital Library
- Di Zang, Zhenliang Chai, Junqi Zhang, Dongdong Zhang, and Jiujun Cheng. 2015. Vehicle license plate recognition using visual attention model and deep learning. Journal of Electronic Imaging 24, 3 (2015), 033001.Google ScholarCross Ref
- Yi Zhu, Xinyu Li, Chunhui Liu, Mohammadreza Zolfaghari, Yuanjun Xiong, Chongruo Wu, Zhi Zhang, Joseph Tighe, R Manmatha, and Mu Li. 2020. A Comprehensive Study of Deep Video Action Recognition. arXiv preprint arXiv:2012.06567(2020).Google Scholar
Recommendations
Deep learning for anomaly detection in multivariate time series: Approaches, applications, and challenges
AbstractAnomaly detection has recently been applied to various areas, and several techniques based on deep learning have been proposed for the analysis of multivariate time series. In this study, we classify the anomalies into three types, ...
Highlights- The methods for anomaly detection on multivariate time series are reviewed.
- The ...
Deep Weakly-supervised Anomaly Detection
KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data MiningRecent semi-supervised anomaly detection methods that are trained using small labeled anomaly examples and large unlabeled data (mostly normal data) have shown largely improved performance over unsupervised methods. However, these methods often focus on ...
Toward Deep Supervised Anomaly Detection: Reinforcement Learning from Partially Labeled Anomaly Data
KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data MiningWe consider the problem of anomaly detection with a small set of partially labeled anomaly examples and a large-scale unlabeled dataset. This is a common scenario in many important applications. Existing related methods either exclusively fit the ...
Comments