Skip to main content
SpringerLink
Log in

Object proposals for salient object segmentation in videos

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Salient object segmentation in videos is generally broken up in a video segmentation part and a saliency assignment part. Recently, object proposals, which are used to segment the image, have had significant impact on many computer vision applications, including image segmentation, object detection, and recently saliency detection in still images. However, their usage has not yet been evaluated for salient object segmentation in videos. Therefore, in this paper, we investigate the application of object proposals to salient object segmentation in videos. In addition, we propose a new motion feature derived from the optical flow structure tensor for video saliency detection. Experiments on two standard benchmark datasets for video saliency show that the proposed motion feature improves saliency estimation results, and that object proposals are an efficient method for salient object segmentation. Results on the challenging SegTrack v2 and Fukuchi benchmark data sets show that we significantly outperform the state-of-the-art.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Note that you cannot know with certainty which of the λ’s is related to the object and which to the background, but you do know that a high λ2 indicates the presence of two local motion vectors.

References

  1. Achanta R, Hemami S, Estrada F, Susstrunk S (2009) Frequency-tuned salient region detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 1597–1604

  2. Azaza A, Van de Weijer J, Douik A, Masana M (2018) Context proposals for saliency detection. Comput Vis Image Underst 174:1–11

  3. Bigun J, Granlund GH, Wiklund J (1991) Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Trans Pattern Anal Mach Intell 13(8):775–790

    Article  Google Scholar 

  4. Borji A, Cheng M-M, Jiang H, Li J (2014) Salient object detection:, A survey, arXiv preprint arXiv., pp 1411–5878

  5. Borji A, Cheng M-M, Jiang H, Li J (2015) Salient object detection: A benchmark. IEEE Trans Image Process, pp 5706–5722

  6. Breiman L (2001) Random forests. Machine learning, pp 5–32

  7. Brox T, Malik J (2010) Object segmentation by long term analysis of point trajectories. European Conference on Computer Vision, pp 282–295

  8. Bruce N, Tsotsos J (2010) Saliency based on information maximization. In: Advances in Neural Information Processing Systems, pp 282–295

  9. Carreira J, Sminchisescu C (2010) Constrained parametric min-cuts for automatic object segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 3241–3248

  10. Fragkiadaki K, Zhang G, Shi J (2012) Video segmentation by tracing discontinuities in a trajectory embedding. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 1846–1853

  11. Frintrop S, Werner T, García GM (2015) Traditional saliency reloaded: a good old model in new shape. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 82–90

  12. Fukuchi K, Miyazato K, Kimura A, Takagi S, Yamato J (2009) Saliency-based video segmentation with graph cuts and sequentially updated priors. In: IEEE International Conference on Multimedia and Expo, pp 638–641

  13. Goferman S, Zelnik-Manor L, Tal A (2012) Context-aware saliency detection. IEEE Trans Pattern Anal Mach Intell 34(10):1915–1926

    Article  Google Scholar 

  14. Itti L, Koch C, Niebur E, et al. (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259

    Article  Google Scholar 

  15. Kim H, Kim Y, Sim J-Y, Kim C-S (2015) Spatiotemporal saliency detection for video sequences based on random walk with restart. IEEE Trans Image Process 24 (8):2552–2564

    Article  MathSciNet  Google Scholar 

  16. Harel J, Koch C, Perona PO (2006) Graph-based visual saliency. In: Advances in Neural Information Processing Systems, pp 545–5524

  17. Han B, Li X, Gao X, Tao D (2012) A biological inspired features based saliency map. In: International Conference on Computing, Networking and Communications, pp 371–375

  18. Itti L, Baldi P (2005) A Principled approach to detecting surprising events in video. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 631–637

  19. Jiang H, Wang J, Yuan Z, Wu Y, Zheng N, Li S (2013) Salient object detection: A discriminative regional feature integration approach. in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2083–2090

  20. Jiang H, Wang J, Yuan Z, Liu T, Zheng N, Li S (2011) Automatic salient object segmentation based on context and shape prior. In: British Machine Vision Conference, pp 2083–2090

  21. Judd T, Ehinger K, Durand F, Torralba A (2009) Learning to predict where humans look. in: IEEE 12th International Conference on Computer Vision, pp 2106–2113

  22. Kim J, Han D, Tai Y-W, Kim J (2014) Salient region detection via high-dimensional color transform. In: Proc IEEE Conf Comput Vis Pattern Recognit, pp 883–890

  23. Krähenbühl P, Koltun V (2014) Geodesic object proposals. European Conference on Computer Vision, pp 725–739

  24. Li F, Kim T, Humayun A, Tsai D, Rehg JM (2013) Video segmentation by tracking many figure-ground segments. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2192–2199

  25. Liu Z, Zhang X, Luo S, Le Meur O (2014) Superpixel-based spatiotemporal saliency detections. IEEE Trans Circuits Syst Video Technol 24(9):1522–1540

    Article  Google Scholar 

  26. Lin X, Casas JR, Pardas M (2017) 3D point cloud segmentation using a fully connected conditional random field. In: 25th European Signal Processing Conference (EUSIPCO), pp 66–70

  27. Lee YJ, Kim J, Grauman K (2011) Key-segments for video object segmentation. In: International Conference on Computer Vision, pp 1995–2002

  28. Lezama Jo, Alahari K, Sivic J, Laptev I (2011) Track to the future: Spatio-temporal video segmentation with long-range motion cues. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 3369–3376

  29. Li Y, Hou X, Koch C, Rehg JM, Yuille Al (2014) The secrets of salient object segmentation. IEEE Conference on Computer Vision and Pattern Recognition, pp 280–287

  30. Lucas BD, Kanade T, et al. (1981) An iterative image registration technique with an application to stereo vision. In: International Joint Conference on Artificial Intelligence, pp 674–679

  31. Mancas M, Riche N, Leroy J, Gosselin B (2011) Abnormal motion selection in crowds using bottom-up saliency. In: IEEE International Conference on Image Processing, pp 229–232

  32. Ma T, Latecki LJ (2012) Maximum weight cliques with mutex constraints for video object segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 670–677

  33. Marat S, Phuoc TH, Granjon L, Guyader N, Pellerin D, Guérin-Dugué A (2009) Modelling spatio-temporal saliency to predict gaze direction for short videos. In: International journal of computer vision, vol 82,3, Springer, p 213

  34. Mauthner T, Possegger HD, Waltner G, Bischof H (2015) Encoding based saliency detection for videos and images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2494–2502

  35. Nah S, Lee KM (2015) Random forest with data ensemble for saliency detection. In: Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, pp 604–607

  36. Oneata D, Revaud J, Verbeek J, Schmid C (2014) Spatio-temporal object detection proposals. In: European Conference on Computer Vision, pp 737–752

  37. Papazoglou A, Ferrari V (2013) Fast object segmentation in unconstrained video. In: Proceedings of the IEEE International Conference on Computer Vision, pp 1777–1784

  38. Rahman A, Houzet D, Pellerin D, Marat S, Guyader N (2011) Parallel implementation of a spatio-temporal visual saliency model. J Real-Time Image Proc 6 (1):3–14

    Article  Google Scholar 

  39. Rahtu E, Kannala J, Salo M, Heikkilä JE (2010) Segmenting salient objects from images and videos. In: European Conference on Computer Vision, pp 366–379

  40. Simoncelli EP, Adelson EH, Heeger DJ (1991) Probability distributions of optical flow. In: Computer Society Conference on Computer Vision and Pattern Recognition, pp 310–315

  41. Singh A, Chu C-HH, Pratt M (2015) Learning to predict video saliency using temporal superpixels. In: International Conference on Pattern Recognition Applications and Methods, pp 201–209

  42. Uijlings JRR, van de Sande KEA, Gevers T, Smeulders AWM (2013) Selective search for object recognition. Int J Comput Vis 104(2):154–171

  43. Van De Weijer J, Gevers T, Smeulders AWM (2006) Robust Photometric invariant features from the color tensor. IEEE Trans Image Process 15(1):118–127

  44. Wang J, Tavakoli H, Laaksonen J (2017) Fixation prediction in videos using unsupervised hierarchical features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 50–57

  45. Wang X, Ma H, Chen X (2016) Geodesic weighted Bayesian model for saliency optimization. Pattern Recogn Lett 75:1–8

    Article  Google Scholar 

  46. Wang W, Shen J, Porikli F (2015) Saliency-aware geodesic video object segmentation. In: Proc IEEE Conf Comput Vis Pattern Recognit, pp 3395–3402

  47. Wang L, Lu H, Ruan Xg, Yang M (2015) Deep networks for saliency detection via local estimation and global search. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 3183–3192

  48. Wang P, Wang J, Zeng G, Feng J, Zha H, Li S (2012) Salient object detection for searched web images via global saliency. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 3194–3201

  49. Yang C, Zhang L, Lu H, Ruan X, Yang M-H (2013) Saliency detection via graph-based manifold ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3166–3173

  50. Zhang D, Javed O, Shah M (2013) Video object segmentation through spatially accurate and temporally dense extraction of primary object regions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 628–635

  51. Zhong S-h, Liu Y, Ren F, Zhang J, Ren T (2013) Video Saliency detection via dynamic consistent Spatio-Temporal attention modelling. In: National Conference of the American Association for Artificial Intelligence, pp 1063–1069

Download references

Author information

Authors and Affiliations

  1. National Engineering School of Sousse, ISITCOM Hammam Sousse, University of Sousse, Sahloul, 4054, Tunisia

    Rahma Kalboussi

  2. National Engineering School of Monastir, University of Monastir, Ibn Eljazzar street, Monastir, 5000, Tunisia

    Aymen Azaza

  3. Computer Vision Center, Edificio O Campus UAB, 08193, Bellaterra (Cerdanyola), Barcelona, Spain

    Joost van de Weijer

  4. National Engineering School of Sousse, University Of Sousse, Pole Technologique de Sousse, Route de Ceinture Sahloul, Sousse, 4054, Tunisia

    Mehrez Abdellaoui & Ali Douik

Authors
  1. Rahma Kalboussi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aymen Azaza
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joost van de Weijer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mehrez Abdellaoui
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ali Douik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rahma Kalboussi.

Ethics declarations

Conflict of interests

Author Rahma Kalboussi declares that she has no conflict of interest. Aymen Azaza has conflict of interest with the Universidad Autonoma de Barcelona where he is a current PhD student (email extension uab.es). Joost van de Weijer has conflict of interest with the Universidad Autonoma de Barcelona where he works (email extension uab.es). Mehrez Abdellaoui and Ali Douik and has conflict of interest with the University of sousse where they work.

Additional information

Human and Animal Rights and Informed Consent

During this project, research has involved no Human Participants and/or Animals. Informed consent was obtained from all individual participants included in this project.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kalboussi, R., Azaza, A., van de Weijer, J. et al. Object proposals for salient object segmentation in videos. Multimed Tools Appl 79, 8677–8693 (2020). https://doi.org/10.1007/s11042-019-07781-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-07781-0

Keywords

Search

Navigation