Abstract
Anomaly detection in surveillance videos aiming to locate anomalous events is a very challenging task, since when training a detector, only normal video samples are available. And thus, most existing approaches address this problem in a semi-supervised way by either predicting or reconstructing the video frames and then compute anomaly scores by comparing the generated frames with reference frames. However, reconstruction-based methods usually lead to mis-detection due to the excessively powerful reconstruction abilities yet the incapable capturing of temporal information, while prediction-based methods are able to perceive temporal information but insufficient to produce realistic future frames. To overcome these problems, we propose a novel Mutuality-Oriented Reconstruction and Prediction Hybrid Network (MORPH-Net) for detecting anomalous events. In the MORPH-Net, a new Mutuality-oriented Training (MO-Training) mechanism is introduced to better combine the advantages of prediction-based models and reconstruction-based models. Compared to traditional single training mechanism or simple fusion mechanism, the MO-Training mechanism can prompt the generator module to produce temporally discriminative and realistic frames which benefit the anomaly detection. The experiments evaluated on three large-scale video surveillance datasets show the efficacy of our method compared with the state-of-the-art approaches.
Similar content being viewed by others
References
Biswas, S., Babu, R.V.: Sparse representation based anomaly detection with enhanced local dictionaries. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 5532–5536. IEEE (2014)
Chen, T., Hou, C., Wang, Z., Chen, H.: Anomaly detection in crowded scenes using motion energy model. Multimed. Tools Appl. 77(11), 14137–14152 (2018)
Chong, Y.S., Tay, Y.H.: Abnormal event detection in videos using spatiotemporal autoencoder. In: International Symposium on Neural Networks, pp. 189–196. Springer (2017)
Cong, Y., Yuan, J., Liu, J.: Sparse reconstruction cost for abnormal event detection. In: CVPR 2011, pp 3449–3456. IEEE (2011)
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 886–893. IEEE (2005)
Dalal, N., Triggs, B., Schmid, C.: Human detection using oriented histograms of flow and appearance. In: European Conference on Computer Vision, pp. 428–441. Springer (2006)
Del Giorno, A., Bagnell, J.A., Hebert, M.: A discriminative framework for anomaly detection in large videos. In: European Conference on Computer Vision, pp. 334–349. Springer (2016)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020)
Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A.K., Davis, L.S.: Learning temporal regularity in video sequences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 733–742 (2016)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes (2013). arXiv preprint arXiv:1312.6114
Lee, S., Kim, H.G., Ro, Y.M.: Stan: spatio-temporal adversarial networks for abnormal event detection. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1323–1327. IEEE (2018)
Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 18–32 (2013)
Liu, W., Luo, W., Lian, D., Gao, S.: Future frame prediction for anomaly detection—a new baseline. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6536–6545 (2018)
Lu, C., Shi, J., Jia, J.: Abnormal event detection at 150 fps in matlab. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2720–2727 (2013)
Luo, W., Liu, W., Gao, S.: Remembering history with convolutional LSTM for anomaly detection. In: 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 439–444. IEEE (2017)
Mahadevan, V., Li, W., Bhalodia, V., Vasconcelos, N.: Anomaly detection in crowded scenes. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1975–1981. IEEE (2010)
Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error (2015). arXiv preprint arXiv:1511.05440
Nguyen, T.N., Meunier, J.: Anomaly detection in video sequence with appearance-motion correspondence. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1273–1283 (2019)
Sabokrou, M., Khalooei, M., Fathy, M., Adeli, E.: Adversarially learned one-class classifier for novelty detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3379–3388 (2018)
Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6479–6488 (2018)
Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6479–6488 (2018)
Tang, Y., Zhao, L., Zhang, S., Gong, C., Li, G., Yang, J.: Integrating prediction and reconstruction for anomaly detection. Pattern Recognit. Lett. 129, 123–130 (2020)
Tudor, I.R., Smeureanu, S., Alexe, B., Popescu, M.: Unmasking the abnormal events in video. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2895–2903 (2017)
Wang, W., Chang, F., Mi, H.: Intermediate fused network with multiple timescales for anomaly detection. Neurocomputing 433, 37–49 (2021)
Zhao, J., Yi, Z., Pan, S., Zhao, Y., Zhao, Z., Su, F., Zhuang, B.: Unsupervised traffic anomaly detection using trajectories. In: CVPR Workshops, pp. 133–140 (2019)
Zhou, J.T., Du, J., Zhu, H., Peng, X., Liu, Y., Goh, R.S.M.: Anomalynet: an anomaly detection network for video surveillance. IEEE Trans. Inf. Forensics Secur. 14(10), 2537–2550 (2019)
Acknowledgements
The project is supported by National Key R&D Program of China (NO.2018YFB1305300), National Natural Science Foundation of China (62176138, 62176136), Shandong Provincial Key Research and Development Program (Major Scientific and Technological Innovation Project ) (NO. 2019JZZY010130, 2020CXGC010207).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Wang, W., Chang, F. & Liu, C. Mutuality-oriented reconstruction and prediction hybrid network for video anomaly detection. SIViP 16, 1747–1754 (2022). https://doi.org/10.1007/s11760-021-02131-w
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-021-02131-w