Skip to main content
SpringerLink
Log in

Multi-scale visualization based on sketch interaction for massive surveillance video data

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Large amount of surveillance video data needs effective and interactive methods to record and utilize. Traditional methods mainly focus on low-level analysis of surveillance video data and lack user interaction. Therefore, there is a strong demand of easy-to-use interaction for efficiently analyzing surveillance video information. In this paper, we propose a multi-scale approach merged on the data flow, objects, and frames to achieve visualization of surveillance video data. Combined with the advantage of sketch interaction, the design of multi-scale structure makes the analysis of surveillance content natural and fluent with annotation of video contents. Extensive user studies demonstrate the effectiveness for facilitating users’ interactive visual analysis in surveillance video exploration and significantly reduce playback time to confirm available information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Babaguchi N, Fujimoto Y, Yamazawa K, Yokoya N (2002) A system for visualization and summarization of omnidirectional surveillance video. In: Multimedia Information Systems. Citeseer, pp 18–27

  2. Bao X, Javanbakhti S, Zinger S, Wijnhoven R et al (2013) Context modeling combined with motion analysis for moving ship detection in port surveillance. J Electron Imaging 22(4):041114

    Article  Google Scholar 

  3. Barnes C, Goldman DB, Shechtman E, Finkelstein A (2010) Video tapestries with continuous temporal zoom. In: ACM Transactions on Graphics (TOG), vol 29. ACM, p 89

  4. Benfold B, Reid I (2011) Stable multi-target tracking in real-time surveillance video. In: Computer Vision & Pattern Recognition

  5. Browne J, Lee B, Carpendale S, Riche N, Sherwood T (2011) Data analysis on interactive whiteboards through sketch-based interaction. In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces. ACM, pp 154–157

  6. Chen L, Shen J, Wang W, Ni B (2015) Video object segmentation via dense trajectories. IEEE Trans Multimed 17(12):2225–2234

    Article  Google Scholar 

  7. Collomosse J, McNeill G, Qian Y (2009) Storyboard sketches for content based video retrieval. In: 2009 IEEE 12th International Conference on Computer Vision. IEEE, pp 245–252

  8. Fang X, Xia Z, Su C, Xu T, Tian Y, Wang Y, Huang T (2013) A system based on sequence learning for event detection in surveillance video. In: 2013 IEEE International Conference on Image Processing. IEEE, pp 3587–3591

  9. Frejlichowski D, Gościewska K, Forczmański P, Hofman R (2015) Application of foreground object patterns analysis for event detection in an innovative video surveillance system. Pattern Anal Applic 18(3):473–484

    Article  MathSciNet  Google Scholar 

  10. Frey BJ, Dueck D (2007) Clustering by passing messages between data points. Science 315(5814):972–976

    Article  MathSciNet  Google Scholar 

  11. Gantz J, Reinsel D (2012) The digital universe in 2020: big data, bigger digital shadows, and biggest growth in the far east. IDC iView: IDC Analyze the Future 2007(2012):1–16

    Google Scholar 

  12. Gelgon M, Bouthemy P (1998) Determining a structured spatio-temporal representation of video content for efficient visualization and indexing. In: European Conference on Computer Vision. Springer, pp 595–609

  13. Goyette N, Jodoin PM, Porikli F, Konrad J, Ishwar P (2012) Changedetection. net: a new change detection benchmark dataset. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. IEEE, pp 1–8

  14. Hu R, James S, Wang T, Collomosse J (2013) Markov random fields for sketch based video retrieval. In: Proceedings of the 3rd ACM Conference on International Conference on Multimedia Retrieval. ACM, pp 279–286

  15. Jang S, Elmqvist N, Ramani K (2015) Motionflow: visual abstraction and aggregation of sequential patterns in human motion tracking data. IEEE Trans Vis Comput Graph 22(1):21– 30

    Article  Google Scholar 

  16. Javanbakhti S, Zinger S, de With P (2014) Context-based region labeling for event detection in surveillance video. In: 2014 International Conference on Information Science, Electronics and Electrical Engineering, vol 1. IEEE, pp 94–98

  17. Jiang F, Wu Y, Katsaggelos AK (2007) Abnormal event detection from surveillance video by dynamic hierarchical clustering. In: 2007 IEEE International Conference on Image Processing, vol 5. IEEE, pp v–145

  18. Liu YJ, Tang K, Joneja A (2005) Sketch-based free-form shape modelling with a fast and stable numerical engine. Comput Graph 29(5):771–786

    Article  Google Scholar 

  19. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  20. Meng J, Yuan J, Yang J, Wang G, Tan YP (2015) Object instance search in videos via spatio-temporal trajectory discovery. IEEE Trans Multimedia 18(1):116–127

    Article  Google Scholar 

  21. Moxley E, Mei T, Manjunath BS (2010) Video annotation through search and graph reinforcement mining. IEEE Trans Multimed 12(3):184–193

    Article  Google Scholar 

  22. Noh S, Jeon M (2012) A new framework for background subtraction using multiple cues. In: Asian Conference on Computer Vision. Springer, pp 493–506

  23. Russell BC, Torralba A, Murphy KP, Freeman WT (2008) Labelme: a database and web-based tool for image annotation. Int J Comput Vis 77(1-3):157–173

    Article  Google Scholar 

  24. Sabirin H, Kim J, Kim M (2011) Graph-based object detection and tracking in h. 264/avc bitstreams for surveillance video. In: 2011 IEEE International Conference on Multimedia and Expo. IEEE, pp 1–6

  25. Sun Z, Liu J (2005) Informal user interfaces for graphical computing. lncs 3784. In: Jiang W, Sun Z (eds) LNCS 3784 (2005). Citeseer, pp 675–682

  26. Verma KK, Kumar P, Tomar A (2015) Analysis of moving object detection and tracking in video surveillance system. In: 2015 2nd International Conference on Computing for Sustainable Global Development (INDIACom). IEEE, pp 1758–1762

  27. Wang H, Ma C (2013) Interactive multi-scale structures for summarizing video content. Sci China Inform Sci 56(5):1–12

    Google Scholar 

  28. Wang J, Zhang G (2011) Video data mining based on k-means algorithm for surveillance video. In: 2011 International Conference on Image Analysis and Signal Processing. IEEE, pp 623–626

  29. Wang M, Hong R, Li G, Zha ZJ, Yan S, Chua TS (2012) Event driven web video summarization by tag localization and key-shot identification. IEEE Trans Multimed 14(4):975–985

    Article  Google Scholar 

  30. Whitaker RT, Mirzargar M, Kirby RM (2013) Contour boxplots: a method for characterizing uncertainty in feature sets from simulation ensembles. IEEE Trans Vis Comput Graph 19(12):2713– 2722

    Article  Google Scholar 

  31. Xu W, Zhang Y, Lu J, Tian Y, Wang J (2011) A framework of simple event detection in surveillance video. In: International Conference on Intelligent Computing and Information Science. Springer, pp 556–561

  32. Yuk JS, Wong KYK, Chung RH, Chow K, Chin FY, Tsang KS (2007) Object-based surveillance video retrieval system with real-time indexing methodology. In: International Conference Image Analysis and Recognition. Springer, pp 626–637

  33. Zhang S, Yu X, Sui Y, Zhao S, Zhang L (2015) Object tracking with multi-view support vector machines. IEEE Trans Multimed 17(3):265–278

    Google Scholar 

  34. Zhang T, Chowdhery A, Bahl PV, Jamieson K, Banerjee S (2015) The design and implementation of a wireless video surveillance system. In: Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, pp 426–438

  35. Zhang T, Liu S, Xu C, Lu H (2012) Mining semantic context information for intelligent video surveillance of traffic scenes. IEEE Trans Ind Inf 9(1):149–160

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Key Research and Development Plan under Grant 2016YFB1001200, and the Natural Science Foundation of China under Grant 61872346.

Author information

Authors and Affiliations

  1. State Key Laboratory of Computer Science and Beijing Key Lab of Human-Computer Interaction, Institute of Software, Chinese Academy of Sciences, Beijing, China

    Zhengming Zhang, Ran Zuo, Cuixia Ma & Hongan Wang

  2. University of Chinese Academy of Sciences, Huairou, China

    Zhengming Zhang, Ran Zuo, Ti Zhou, Cuixia Ma & Hongan Wang

  3. Beijing AIMan Data Technology Co., Ltd., Beijing, China

    Rui Guo

  4. Nanjing Software Institute, Nanjing, China

    Yanfeng Li

  5. Beijing University of Posts and Telecommunications, Beijing, China

    Han Xue

Authors
  1. Zhengming Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ran Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rui Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yanfeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ti Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Han Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Cuixia Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Hongan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cuixia Ma.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Z., Zuo, R., Guo, R. et al. Multi-scale visualization based on sketch interaction for massive surveillance video data. Pers Ubiquit Comput 25, 1027–1037 (2021). https://doi.org/10.1007/s00779-019-01281-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-019-01281-6

Keywords

Search

Navigation