Skip to main content
Springer Nature Link
Log in

Spatiotemporal salient object detection by integrating with objectness

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper proposes a novel spatiotemporal salient object detection method by integrating saliency and objectness, for videos with complicated motion and complex scenes. The initial salient object detection result is first built upon both saliency map and objectness map. Afterwards, the region size of salient object is adjusted to obtain the frame-wise salient object detection result by iteratively updating the object probability map, which is the combination of saliency map and objectness map. Finally, in order to enhance the temporal coherence, the sequence-level refinement is performed to generate the final salient object detection result. Experimental results on public benchmark datasets demonstrate that the proposed method consistently outperforms the state-of-the-art salient object detection methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Bao L, Lu J, Li Y, Shi Y (2015) A saliency detection model using shearlet transform. Multimed Tools Appl 74(11):4045–4058

    Article  Google Scholar 

  2. Bao L, Zhang X, Zheng Y, Li Y (2016) Video saliency detection using 3D shearlet transform. Multimed Tools Appl 75(13):7761–7778

    Article  Google Scholar 

  3. Borji A, Cheng M-M, Jiang H, Li J (2014) Salient Object Detection: A Survey. arXiv preprint arXiv:1411.5878

  4. Borji A, Cheng M-M, Jiang H, Li J (2015) Salient Object Detection: A Benchmark. IEEE Trans Image Process 24(12):5706–5722

    Article  MathSciNet  Google Scholar 

  5. Chan AB, Vasconcelos N (2007) Classifying video with kernel dynamic textures. In: Proc. IEEE CVPR, pp. 1–6

  6. Cheng W-H, Wang C-W, Wu J-L (2007) Video adaptation for small display based on content recomposition. IEEE Trans Circ Syst Video Technol 17(1):43–58

    Article  Google Scholar 

  7. Du H, Liu Z, Jiang J, Shen L (2013) Stretchability-aware block scaling for image retargeting. J Vis Commun Image Represent 24(4):499–508

    Article  Google Scholar 

  8. Fang Y, Lin W, Chen Z, Tsai C-M, Lin C-W (2014) A video saliency detection model in compressed domain. IEEE Trans Circ Syst Video Technol 24(1):27–38

    Article  Google Scholar 

  9. Felzenszwalb PF, Huttenlocher DP (2004) Efficient graph-based image segmentation. Int J Comput Vis 59(2):167–181

    Article  Google Scholar 

  10. Gopalakrishnan V, Hu Y, Rajan D (2010) Random walks on graphs for salient object detection in images. IEEE Trans Image Process 19(12):3232–3242

    Article  MathSciNet  MATH  Google Scholar 

  11. Guo C, Zhang L (2010) A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression. IEEE Trans Image Process 19(1):185–198

    Article  MathSciNet  MATH  Google Scholar 

  12. Huang J, Li Z (2009) Image trimming via saliency region detection and iterative feature matching. In: Proc. IEEE ICME, pp. 1322–1325

  13. Ko BC, Nam JY (2006) Object-of-interest image segmentation based on human attention and semantic region clustering. J Opt Soc Am A 23(10):2462–2470

    Article  Google Scholar 

  14. Lampert CH, Blaschko MB, Hofmann T (2009) Efficient subwindow search: A branch and bound framework for object localization. IEEE Trans Patt Anal Mach Intell 31(12):2129–2142

    Article  Google Scholar 

  15. Li J, Liu Z (2015) Object detection based on visual saliency map and objectness. J Comput Appl 35(12):3560–3564

    Google Scholar 

  16. Li Z, Qin S, Itti L (2011) Visual attention guided bit allocation in video compression. Image Vis Comput 29(1):1–14

    Article  Google Scholar 

  17. Li F, Kim T, Humayun A, Tsai D, Rehg JM (2013) Video segmentation by tracking many figure-ground segments. In: Proc. IEEE ICCV, pp. 2192–2199

  18. Li J, Liu Z, Zhang X, Le Meur O, Shen L (2015) Spatiotemporal saliency detection based on superpixel-level trajectory. Signal Process Image Commun 38:100–144

    Article  Google Scholar 

  19. Lin R-S, Liu C-B, Yang M-H, Ahuja N, Levinson S (2006) Learning nonlinear manifolds from time series. Proc. IEEE ECCV, In, pp 245–256

    Google Scholar 

  20. Liu C (2009) Beyond pixels: Exploring new representations and applications for motion analysis. Ph.D. dissertation, Massachusetts Inst. Technol., Cambridge, MA, USA

  21. Liu F, Gleicher M (2005) Automatic image retargeting with fisheye-view warping. In: Proc. ACM Symp. User Interface Software and Technology, pp. 153–162

  22. Liu T, Sun J, Zheng N, Tang X, Shum HY (2007) Learning to detect a salient object. In: Proc. IEEE CVPR, pp. 1–8

  23. Liu Q, Han T, Sun Y, Chu Z, Shen B (2013) A two step salient objects extraction framework based on image segmentation and saliency detection. Multimed Tools Appl 67(1):231–247

    Article  Google Scholar 

  24. Liu Z, Zou W, Le Meur O (2014) Saliency tree: a novel saliency detection framework. IEEE Trans Image Process 23(5):1937–1952

    Article  MathSciNet  MATH  Google Scholar 

  25. Liu Z, Zhang X, Luo S, Le Meur O (2014) Superpixel-based spatiotemporal saliency detection. IEEE Trans Circ Syst Video Technol 24(9):1522–1540

    Article  Google Scholar 

  26. Liu Z, Li J, Ye L, Sun G, Shen L (2016) Saliency Detection for Unconstrained Videos Using Superpixel-level Graph and Spatiotemporal Propagation. IEEE Trans Circ Syst Video Technol. https://doi.org/10.1109/TCSVT.2016.2595324

  27. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proc. IEEE CVPR, pp. 3431–3440

  28. Luo Y, Yuan J, Xue P, Tian Q (2010) Saliency density maximization for object detection and localization. In: Proc. ACCV, pp. 396–408

  29. Luo Y, Yuan J, Lu J (2016) Finding spatio-temporal salient paths for video objects discovery. J Vis Commun Image Represent 38:45–54

    Article  Google Scholar 

  30. Ma YF, Zhang HJ (2003) Contrast-based image attention analysis by using fuzzy growing. In: Proc. ACM Int. Conf. Multimedia, pp. 374–381

  31. Mertsching B, Bollmann M, Hoischen R, Schmalz S (1999) The neural active vision system NAVIS. Handbook Comput Vision Appl 3:543–568

    Google Scholar 

  32. Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 9(1):62–66

    Article  MathSciNet  Google Scholar 

  33. Peng H, Li B, Ling H, Hu W, Xiong W, Maybank SJ (2017) Salient object detection via structured matrix decomposition. IEEE Trans Pattern Anal Mach Intell 39(4):818–832

    Article  Google Scholar 

  34. Perazzi F, Pont-Tuset J, Mc Williams B, Gool LV, Gross M, Sorkine-Hornung A (2016) A benchmark dataset and evaluation methodology for video object segmentation. In: Proc. IEEE CVPR, pp. 724–732

  35. Shamir A, Avidan S (2009) Seam carving for media retargeting. Comm ACM 52(1):77–85

    Article  Google Scholar 

  36. Shen L, Liu Z, Zhang Z (2013) A novel H.264 rate control algorithm with consideration of visual attention. Multimed Tools Appl 63(3):709–727

    Article  Google Scholar 

  37. Shi R, Liu Z, Du H, Zhang X, Shen L (2012) Region diversity maximization for salient object detection. IEEE Sign Process Lett 19(4):215–218

    Article  Google Scholar 

  38. Tian Y, Li J, Yu S, Huang T (2015) Learning complementary saliency priors for foreground object segmentation in complex scenes. Int J Comput Vis 111(2):153–170

    Article  Google Scholar 

  39. Valenti R, Sebe N, Gevers T (2009) Image saliency by isocentric curvedness and color. In: Proc. IEEE ICCV, pp. 2185–2192

  40. Wang W, Shen J, Porikli F (2015) Saliency-aware geodesic video object segmentation. In: Proc. IEEE CVPR, pp. 3395–3402

  41. Wang W, Shen J, Shao L (2015) Consistent video saliency using local gradient flow optimization and global refinement. IEEE Trans Image Process 24(12):4185–4196

    Article  MathSciNet  Google Scholar 

  42. Wu B, Xu L (2014) Integrating bottom-up and top-down visual stimulus for saliency detection in news video. Multimed Tools Appl 73(3):1053–1075

    Article  Google Scholar 

  43. Wu T, Liu Z, Li J (2016) Spatiotemporal saliency detection using border connectivity. In: Eight International Conference on Digital Image Processing, article 1003344

  44. Yan C, Zhang Y, Xu J, Dai F, Li L, Dai Q, Wu F (2014) A highly parallel framework for HEVC coding unit partitioning tree decision on many-core processors. IEEE Sign Process Lett 21(5):573–576

    Article  Google Scholar 

  45. Yan C, Zhang Y, Xu J, Dai F, Li L, Dai Q, Wu F (2014) Efficient parallel framework for HEVC motion estimation on many-core processors. IEEE Trans Circ Syst Video Technol 24(12):2077–2089

    Article  Google Scholar 

  46. Yan Y, Nie F, Li W, Gao C, Yang Y, Xu D (2016) Image classification by cross-media active learning with privileged information. IEEE Trans Multimed 18(12):2494–2502

    Article  Google Scholar 

  47. Ye L, Liu Z, Li L (2015) Evaluation on fusion of saliency and objectness for salient object segmentation. In: Proc. ICIMCS, pp. 1–4

  48. You X, Guo W, Yu S, Yu S, Li K, Príncipe JC, Tao D (2016) Kernel learning for dynamic texture synthesis. IEEE Trans Image Process 25(10):4782–4795

    Article  MathSciNet  Google Scholar 

  49. Yuan Z, Lu T, Huang Y, Wu D, Yu H (2012) Addressing visual consistency in video retargeting: A refined homogeneous approach. IEEE Trans Circ Syst Video Technol. 22(6):890–903

    Article  Google Scholar 

  50. Zitnick CL, Dollár P (2014) Edge boxes: Locating object proposals from edges. In: European Conference on Computer Vision, pp. 391–405

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant No. 61471230 and No. 61601278, and by the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.

Author information

Authors and Affiliations

  1. Shanghai Institute for Advanced Communication and Data Science, Shanghai University, Shanghai, , 200444, China

    Tongbao Wu, Zhi Liu, Xiaofei Zhou & Kai Li

  2. School of Communication and Information Engineering, Shanghai University, Shanghai, , 200444, China

    Tongbao Wu, Zhi Liu, Xiaofei Zhou & Kai Li

Authors
  1. Tongbao Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Zhi Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Xiaofei Zhou
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Kai Li
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Zhi Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, T., Liu, Z., Zhou, X. et al. Spatiotemporal salient object detection by integrating with objectness. Multimed Tools Appl 77, 19481–19498 (2018). https://doi.org/10.1007/s11042-017-5334-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5334-1

Keywords