Skip to main content
SpringerLink
Log in

Abnormal event detection in tourism video based on salient spatio-temporal features and sparse combination learning

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

With the booming development of tourism, travel security problems are becoming more and more prominent. Congestion, stampedes, fights and other tourism emergency events occurred frequently, which should be a wake-up call for tourism security. Therefore, it is of great research value and application prospect to real-time monitor tourists and detect abnormal events in tourism surveillance video by using computer vision and video intelligent processing technology, which can realize the timely forecast and early warning of tourism emergencies. At present, although most of the video-based abnormal event detection methods work well in simple scenes, there are often problems such as low detection rate and high false positive rate in complex motion scenarios, and the detection of abnormal events can’t be processed in real time. To tackle these issues, we propose an abnormal event detection model in tourism video based on salient spatio-temporal features and sparse combination learning, which has good robustness and timeliness in complex motion scenarios and can be adapted to real-time anomaly detection in practical applications. Specifically, spatio-temporal gradient model is combined with foreground detection to extract 3D gradient features on the foreground target of video sequence as the salient spatio-temporal features, which can eliminate the interference of the background. Sparse combination learning algorithm is used to establish the abnormal event detection model, which can realize the real-time detection of abnormal events. In addition, we construct a new ScenicSpot dataset with 18 video clips (5964 frames) containing both normal and abnormal events. The experimental results on ScenicSpot dataset and two standard benchmark datasets show that our method can realize the automatic detection and recognition of tourists’ abnormal behavior, and has better performance compared with the classical methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Figure 18
Figure 19

Similar content being viewed by others

References

  1. Benezeth, Y., Jodoin, P.M., Saligrama, V., Rosenberger, C.: Abnormal events detection based on spatio-temporal co-occurences. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2458–2465 (2009). https://doi.org/10.1109/CVPRW.2009.5206686

  2. Chen, H., Zhao, X., Wang, T., Tan, M., Sun, S.: Spatial-temporal context-aware abnormal event detection based on incremental sparse combination learning. In: 2016 12th World Congress on Intelligent Control and Automation (WCICA), pp. 640–644 (2016). https://doi.org/10.1109/WCICA.2016.7578533

    Chapter  Google Scholar 

  3. Chen, T., Hou, C., Wang, Z., Chen, H.: Anomaly detection in crowded scenes using motion energy model. Multimedia Tools and Applications. 3, (2017). https://doi.org/10.1007/s11042-017-5020-3

  4. Colque, R.V.H.M., Schwartz, W.R.: Histograms of optical flow orientation and magnitude to detect anomalous events in videos. IEEE Transactions on Circuits and Systems for Video Technology. 99 (2017). https://doi.org/10.1109/SIBGRAPI.2015.21

  5. Cong, Y., Yuan, J., Liu, J.: Abnormal event detection in crowded scenes using sparse representation. Pattern Recogn. 46(7), 1851–1864 (2013). https://doi.org/10.1016/j.patcog.2012.11.021

    Article  Google Scholar 

  6. Cui, X., Liu, Q., Gao, M., Metaxas, D.N.: Abnormal detection using interaction energy potentials. The 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 42(7). 3161–3167 (2011). https://doi.org/10.1109/CVPR.2011.5995558

  7. Cui, J., Liu, W., Xing, W.: Crowd behaviors analysis and abnormal detection based on surveillance data. J. Vis. Lang. Comput. 25(6), 628–636 (2014). https://doi.org/10.1016/j.jvlc.2014.10.032

    Article  Google Scholar 

  8. Du, D., Qi, H., Huang, Q., Zeng, W., Zhang, C.: Abnormal event detection in crowded scenes based on structural multi-scale motion interrelated patterns. IEEE International Conference on Multimedia and Expo (ICME). 1–6 (2013). https://doi.org/10.1109/ICME.2013.6607499

  9. Feng, Y., Yuan, Y., Lu, X.: Learning deep event models for crowd anomaly detection. Neurocomputing. 219, 548–556 (2017). https://doi.org/10.1016/j.neucom.2016.09.063

    Article  Google Scholar 

  10. Gao, L., Song, J., Liu, X., Shao, J., Liu, J., Shao, J.: Learning in high-dimensional multimedia data: the state of the art. Multimedia Systems. 23(3), 1–11 (2017). https://doi.org/10.1007/s00530-015-0494-1

    Article  Google Scholar 

  11. Gao, L., Guo, Z., Zhang, H., Xu, X., Shen, H.T.: Video captioning with attention-based LSTM and semantic consistency. IEEE Transactions on Multimedia. 19(9), 2045–2055 (2017). https://doi.org/10.1109/TMM.2017.2729019

    Article  Google Scholar 

  12. Gogoi, P., Bhattacharyya, D.K., Borah, B., Kalita, J.K.: A survey of outlier detection methods in network anomaly identification. Comput. J. 54(4), 570–588 (2011). https://doi.org/10.1093/comjnl/bxr026

    Article  Google Scholar 

  13. Hu, D.H., Zhang, X.X., Yin, J., Zheng, V.W., Yang, Q.: Abnormal activity recognition based on HDP-HMM models. In: Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI), pp. 1715–1720 (2009)

    Google Scholar 

  14. Huang, D., Hu, W., Wu, X., et al.: The algorithm of video foreground extraction via improved single gauss model and merge of broken targets. J. Signal Process. 3, 299–307 (2015). https://doi.org/10.3969/j.issn.1003-0530.2015.03.007

    Google Scholar 

  15. Jolliffe, I.T., Cadima, J.: Principal component analysis: a review and recent developments. Phil. Trans. R. Soc. A. 374(2065), 20150202 (2016). https://doi.org/10.1098/rsta.2015.0202

    Article  MathSciNet  MATH  Google Scholar 

  16. Kong, L., Guo, L., Wang, Q., Han, Y.: Improvement of linear filter in image denoising. In: International Conference on Intelligent Earth Observing and Applications, Pp. 98083F (2015). https://doi.org/10.1117/12.2207241

    Google Scholar 

  17. Kratz, L., Nishino, K.: Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1446–1453 (2009). https://doi.org/10.1109/CVPR.2009.5206771

  18. Leyva, R., Sanchez, V., Li, C.T.: Video anomaly detection with compact feature sets for online performance. IEEE Trans. Image Process. 26, 99–3478 (2017). https://doi.org/10.1109/TIP.2017.2695105

    Article  MathSciNet  MATH  Google Scholar 

  19. Li, A., Miao, Z., Cen, Y., Liang, Q.: Abnormal event detection based on sparse reconstruction in crowded scenes. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1786–1790 (2016). https://doi.org/10.1109/ICASSP.2016.7471984

  20. Li, X., Zhou, Z., Chen, L., Gao, L.: Residual attention-based LSTM for video captioning. World Wide Web: Internet and Web Information Systems. 9, 1–16 (2018). https://doi.org/10.1007/s11280-018-0531-z

    Google Scholar 

  21. Liu, Z., Feng, X., Zhang, J.: Action recognition based on deep convolution neural network and depth sequence. Journal of Chongqing University (Natural Science Edition). 40(11), 99–106 (2017). https://doi.org/10.11835/j.issn.1000-582X.2017.11.012

    Google Scholar 

  22. Lu, C., Shi, J., Jia, J.: Abnormal event detection at 150 FPS in MATLAB. In: IEEE International Conference on Computer Vision (ICCV), pp. 2720–2727 (2014). https://doi.org/10.1109/ICCV.2013.338

    Google Scholar 

  23. Mahadevan, V., Li, W., Bhalodia, V., Vasconcelos, N.: Anomaly detection in crowded scenes. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1975–1981 (2010). https://doi.org/10.1109/CVPR.2010.5539872

  24. Mehran, R., Oyama, A., Shah, M.: Abnormal crowd behavior detection using social force model. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). 935–942 (2009). https://doi.org/10.1109/CVPR.2009.5206641

  25. Meng, L.I., Chen, K., Guo, C., Fei, L.I., Peipei, J.I.: Abnormal crowd event detection by fusing saliency information and social force model. Opto-Electron. Eng. (2016)

  26. Miao, Y., Song, J.: Abnormal event detection based on SVM in video surveillance. Advanced Research and Technology in Industry Applications. 1379–1383 (2014). https://doi.org/10.1109/WARTIA.2014.6976540

  27. Mittal, S., Prasad, T., Saurabh, S., Fan, X., Shin, H.: Pedestrian detection and tracking using deformable part models and Kalman filtering. In: Soc Design Conference, 10(7), pp. 960–966 (2013). doi: https://doi.org/10.1109/ISOCC.2012.6407106

    Google Scholar 

  28. Nallaivarothayan, H., Fookes, C., Denman, S., Sridharan, S.: An MRF based abnormal event detection approach using motion and appearance features. IEEE International Conference on Advanced Video and Signal Based Surveillance. 343–348 (2014). https://doi.org/10.1109/AVSS.2014.6918692

  29. Pathan, S.S., Al-Hamadi, A., Michaelis, B.: Using conditional random field for crowd behavior analysis. In: Asian Conference on Computer Vision (ACCV). 6468, 370–379 (2010)

    Google Scholar 

  30. Pennisi, A., Bloisi, D.D., Locchi, L.: Online real-time crowd behavior detection in video sequences. Comput. Vis. Image Underst. 144, 166–176 (2016). https://doi.org/10.1016/j.cviu.2015.09.010

    Article  Google Scholar 

  31. Rao, A.S., Gubbi, J., Marusic, S., Palaniswami, M.: Crowd event detection on optical flow manifolds. IEEE Transactions on Cybernetics. 46(7), 1524–1537 (2016). https://doi.org/10.1109/TCYB.2015.2451136

    Article  Google Scholar 

  32. Ren, H., Moeslund, T.B.: Abnormal event detection using local sparse representation. IEEE International Conference on Advanced Video and Signal Based Surveillance. 125–130 (2014). https://doi.org/10.1109/AVSS.2014.6918655

  33. Shen, Y., Wang, X.: Video moving target detection method based on background subtraction and interframe difference method. Automation & Instrumentation. 4, 122–124 (2017). https://doi.org/10.14016/j.cnki.1001-9227.2017.04.122

    Google Scholar 

  34. Wali, A., Alimi, A.M.: Event detection from video surveillance data based on optical flow histogram and high-level feature extraction. International Workshop on Database and Expert Systems Application. 221–225 (2009). https://doi.org/10.1109/DEXA.2009.81

  35. Wang, T., Snoussi, H.: Histograms of optical flow orientation for visual abnormal events detection. IEEE Ninth International Conference on Advanced Video and Signal-Based Surveillance. 13–18 (2012). https://doi.org/10.1109/AVSS.2012.39

  36. Wang, J., Schweitzer, J., Tilmann, F., White, R.S., Soosalu, H.: Application of the multichannel wiener filter to regional event detection using NORSAR seismic-array data. Bull. Seismol. Soc. Am. 101(6), 2887–2896 (2011). https://doi.org/10.1785/0120110003

    Article  Google Scholar 

  37. Wang, S.M., Fang, L.Y., Deng, F.: Research on the evaluation model of urban tourism management efficiency with uncertain linguistic information. Journal of Control Science and Engineering. 2, 12–14 (2014). https://doi.org/10.1155/2014/582454

    MATH  Google Scholar 

  38. Wang, M., Li, X., Chen, Q., et al.: Surveillance event detection based on CNN. Acta Automat. Sin. 42(6), 892–903 (2016). https://doi.org/10.16383/j.aas.2016.c150729

    Google Scholar 

  39. Wang, C., Yao, H., Sun, X.: Anomaly detection based on spatio-temporal sparse representation and visual attention analysis. Multimedia Tools and Applications. 76, 1–17 (2016). https://doi.org/10.1007/s11042-015-3199-8

    Google Scholar 

  40. Wang, X., Gao, L., Wang, P., Sun, X., Liu, X.: Two-stream 3D convNet fusion for action recognition in videos with arbitrary size and length. IEEE Transactions on Multimedia. PP. 644(99), 1–1 (2017). https://doi.org/10.1109/TMM.2017.2749159

    Google Scholar 

  41. Wang, X., Gao, L., Song, J., Zhen, X., Sebe, N., Shen, H.T.: Deep appearance and motion learning for egocentric activity recognition. Neurocomputing. 275, 438–447 (2018). https://doi.org/10.1016/j.neucom.2017.08.063

    Article  Google Scholar 

  42. Wen, Y., Du, J., Lee, J.M.: Abnormal event detection based on social force model combined with crowd violent flow. International Conference on Cloud Computing and Intelligence Systems. 440–446 (2016). https://doi.org/10.1109/CCIS.2016.7790299

  43. Wriggers, W., Stafford, K.A., Shan, Y., Piana, S., Maragakis, P., Lindorff-Larsen, K., Miller, P.J., Gullingsrud, J., Rendleman, C.A., Eastwood, M.P., Dror, R.O., Shaw, D.E.: Automated event detection and activity monitoring in long molecular dynamics simulations. J. Chem. Theory Comput. 5(10), 2595–2605 (2009)

    Article  Google Scholar 

  44. Wu, C., Li, M., Liu, M., Zheng, Z., Zhang, Y.: Adaptive motion detection based on median background model. Journal of Shenyang Jianzhu University. (2008)

  45. Wu, X., Guo, H., Li, N., et al.: Survey on the video-based abnormal event detection in crowd scenes. Journal of Electronic Measurement and Instrument. 28(6), 575–584 (2014). https://doi.org/10.13382/j.jemi.2014.06.001

    Google Scholar 

  46. Xing, H.U., Shiqiang, H.U., Luo, L., Guoxiang, L.I.: Abnormal event detection in crowded scenes via bag-of-atomic-events-based topic model. Turk. J. Electr. Eng. Comput. Sci. 24, 2638–2653 (2016). https://doi.org/10.3906/elk-1310-191

    Article  Google Scholar 

  47. Xu, D., Ricci, E., Yan, Y., et al.: Detecting anomalous events in videos by learning deep representations of appearance and motion. Learning deep representations of appearance and motion for anomalous event detection. arXiv preprint arXiv. 1510, 01553–01127 (2015). https://doi.org/10.1016/j.cviu.2016.10.010

    Google Scholar 

  48. Yang, H., Cao, Y., Wu, S., Lin, W.: Abnormal crowd behavior detection based on local pressure model. In: Signal and Information Processing Association Summit and Conference (APSIPA ASC), pp. 1–4 (2014)

    Google Scholar 

  49. Ye, R., Li, X.: Collective representation for abnormal event detection. J. Comput. Sci. Technol. 32(3), 470–479 (2017). https://doi.org/10.1007/s11390-017-1737-8

    Article  MathSciNet  Google Scholar 

  50. Yin, C., Xiang, J.Y., Han, J.D.: Small target detection based on mean background model in IR images. Infrared Technology. (2004)

  51. Yong, S.C., Yong, H.T.: Abnormal event detection in videos using spatiotemporal autoencoder. International Symposium on Neural Networks. 189–196 (2017). https://doi.org/10.1007/978-3-319-59081-3_23

  52. Yu, B., Liu, Y., Sun, Q.: A content-adaptively sparse reconstruction method for abnormal events detection with low-rank property. IEEE Transactions on Systems Man and Cybernetics Systems. 99, 1–13 (2016). https://doi.org/10.1109/TSMC.2016.2638048

    Google Scholar 

  53. Yu, Y., Shen, W., Huang, H., Zhang, Z.: Abnormal event detection in crowded scenes using two sparse dictionaries with saliency. Journal of Electronic Imaging. 26(3), 033013 (2017). https://doi.org/10.1117/1.JEI.26.3.033013

    Article  Google Scholar 

  54. Zhang, D., Gaticaperez, D., Bengio, S., Mccowan, I.: Semi-supervised adapted HMMs for unusual event detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). 1, 611–618 (2005). https://doi.org/10.1109/CVPR.2005.316

    Google Scholar 

  55. Zhang, R., Zhou, M., Gong, X., He, X., Qian, W., Qin, S., Zhou, A.: Detecting anomaly in data streams by fractal model. World Wide Web. 18(5), 1419–1441 (2015). https://doi.org/10.1007/s11280-014-0296-y

    Article  Google Scholar 

  56. Zhang, Z., Liu, S., Zhang, Z.: Consistent sparse representation for abnormal event detection. IEICE Trans. Inf. Syst. E98.D(10), 1866–1870 (2015). https://doi.org/10.1587/transinf.2015EDL8113

    Article  Google Scholar 

  57. Zhong, C.: Xu, G.: movement pedestrian detection method combined with foreground subtraction and deep learning. Computer and digital. Engineering. 44(12), 2396–2399 (2016). https://doi.org/10.3969/j.issn.1672-9722.2016.12.023

    Google Scholar 

  58. Zhou, S., Shen, W., Zeng, D., Fang, M., Wei, Y., Zhang, Z.: Spatial-temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Processing Image Communication. 47, 358–368 (2016). https://doi.org/10.1016/j.image.2016.06.007

    Article  Google Scholar 

  59. Zhu, J., Jiang, W., Liu, A., Liu, G., Zhao, L.: Effective and efficient trajectory outlier detection based on time-dependent popular route. World Wide Web. 20(1), 111–134 (2017). https://doi.org/10.1007/s11280-016-0400-6

    Article  Google Scholar 

  60. Zhu, Y., Zhang, X., Wang, R., Zheng, W., Zhu, Y.: Self-representation and PCA embedding for unsupervised feature selection. World Wide Web. 1, 1–14 (2017). https://doi.org/10.1007/s11280-017-0497-2

    Google Scholar 

  61. Zou, Y.H., Guo, C.S.: Video abnormal event detection based on HMM cascaded with LDA. In: Journal of Hangzhou Dianzi University (2013)

    Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China (No. 61320106006, No. 61502042, No. 61532006, No. 61772083) and the Fundamental Research Funds for the Central University (No. 2017RC39).

Author information

Authors and Affiliations

  1. Beijing Key Lab of Intelligent Telecommunication Software and Multimedia, School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, 100876, China

    Yue Geng, Junping Du & Meiyu Liang

Authors
  1. Yue Geng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junping Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meiyu Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junping Du.

Additional information

Guest Editors: Jingkuan Song, Shuqiang Jiang, Elisa Ricci, and Zi Huang

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Special Issue on Deep vs. Shallow: Learning for Emerging Web-scale Data Computing and Applications

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Geng, Y., Du, J. & Liang, M. Abnormal event detection in tourism video based on salient spatio-temporal features and sparse combination learning. World Wide Web 22, 689–715 (2019). https://doi.org/10.1007/s11280-018-0603-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-018-0603-0

Keywords

Search

Navigation