Skip to main content
Springer Nature Link
Log in

Motion cues and saliency based unconstrained video segmentation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The segmentation of moving objects become challenging when the object motion is small, the shape of object changes, and there is global background motion in unconstrained videos. In this paper, we propose a fully automatic, efficient, fast and composite framework to segment the moving object on the basis of saliency, locality, color and motion cues. First, we propose a new saliency measure to predict the potential salient regions. In the second step, we use the RANSAC homography and optical flow to compensate the background motion and get reliable motion information, called motion cues. Furthermore, the saliency information and motion cues are combined to get the initial segmented object (seeded region). A refinement is performed to remove the unwanted noisy details and expand the seeded region to the whole object. Detailed experimentation is carried out on challenging video benchmarks to evaluate the performance of the proposed method. The results show that the proposed method is faster and performs better than state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Bai X, Wang J, Simons D, Sapiro G (2009) Video snapcut: robust video object cutout using localized classifiers. ACM Transactions on Graphics, 28(3)

  2. Banica D, Agape A, Ion A, Sminchisescu C (2013) Video object segmentation by salient segment chain composition. IEEE Conference on Computer Vision Workshop

  3. Brox T, Malik J (2010) Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans Pattern Anal Mach Intell 33(3):500–513

    Article  Google Scholar 

  4. Brox T, Malik J (2010) Object segmentation by long term analysis of point trajectories. In: European Conference on Computer Vision

  5. Bugeau A, Perez P (2007) Detection and segmentation of moving objects in highly dynamic scenes. IEEE Conference on Computer Vision and Pattern Recognition, 1–8

  6. Chiu W, Fritz M (2013) Multi-class video co-segmentation with a generative multi-video model. IEEE Conference on Computer Vision and Pattern Recognition

  7. Chockalingam P, Pradeep N, Birchfield S (2009) Adaptive fragments-based tracking of non-rigid objects using level sets. In: BMVC, pp 1530–1537

  8. Elgammal A, Duraiswami R, Harwood D, Davis L (2002) Background and foreground modeling using nonparametric kernel density estimation for visual surveillance. Proc IEEE 90(7):1151–1163

    Article  Google Scholar 

  9. Faktor A, Irani M (2014) Video segmentation by non-local consensus voting. Proceedings of the British Machine Vision Conference

  10. Fischler M, Bolles R (1981) Random sample consensus: a paradigm for model fitting with application to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  11. Fragkiadaki K, Arbelaez P, Felsen P, Malik J (2015) Learning to segment moving objects in videos. IEEE Conference on Computer Vision and Pattern Recognition, 4083–4090

  12. Galasso F, Nagaraja NS, Cardenas TJ, Brox T, Schiele B (2013) A unified video segmentation benchmark: Annotation, metrics and analysis. IEEE Conference on Computer Vision

  13. Gelgon M, Bouthemy P (1997) A region-level graph labeling approach to motion-based segmentation. IEEE Conference on Computer Vision and Pattern Recognition, 514–519

  14. Grundmann M, Kwatra V, Han M, Essa I (2010) Efficient hierarchical graph based video segmentation. IEEE Conference on Computer Vision and Pattern Recognition

  15. Han M, Xu W, Gong Y (2006) Video object segmentation by motion-based sequential feature clustering. In: ACM Multimedia, pp 773–782

  16. Hartley R, Zisserman A (2004) Multiple View Geometry in Computer Vision. Cambridge University Press second edition

  17. Heikkila M, Pietikainen M (2006) A texture-based method for modeling the background and detecting moving objects. IEEE Trans Pattern Anal Mach Intelligence 28(4):657–662

    Article  Google Scholar 

  18. I2R (2010) Efficient hierar chicalgraph-based video segmentation. IEEE Conference on Computer Vision and Pattern Recognition, 2141–2148

  19. Jojic B, Kannan A (2003) Learning appearance and transparency manifolds of occluded objects in layers. IEEE Conference on Computer Vision and Pattern Recognition

  20. Joulin A, Bach F, Ponce J (2012) Multi-class cosegmentation. IEEE Conference on Computer Vision and Pattern Recognition

  21. KaewTraKulPong P, Bowden R (2001) An improved adaptive background mixture model for real-time tracking with shadow detection. Proceedings of the 2nd European Workshop Advanced Video Based Surveillance Systems 1:1–5

    Google Scholar 

  22. Khan S, Shah M (2001) Object based segmentation of video using color, motion and spatial information. IEEE Conference on Computer Vision and Pattern Recognition, 746–751

  23. Khan A, Ullah J, Jaffar M, Chai T (2014) Color imagesegmentation: A novel spatial fuzzy genetic algorithm. SIViP, Springer 8(7):1233–1243

    Article  Google Scholar 

  24. Lee Y, Kim J, Grauman K (2011) Key-segments for video object segmentation. IEEE Conference on Computer Vision, 1995–2002

  25. Li L, Huang W, Gu I, Tian Q (2003) Foreground object detection from videos containing complex background. ACM International Conference on Multimedia, 2–10

  26. Li F, Kim T, Humayun A, Tsai D, Rehg J (2013) Video segmentation by tracking many figure-ground segments. IEEE Conference on Computer Vision

  27. Liu C (2009) Beyond pixels: Exploring new representations and applications for motion analysis. In: Doctoral Thesis. Massachusetts Institute of Technology

  28. Liu F, Gleicher M (2009) Learning color and locality cues for moving object detection and segmentation. IEEE Conference on Computer Vision and Pattern Recognition

  29. Liu C, Yuen J, Russell B, Torralba A (2009) Labelme video: Building a video database with human annotations. IEEE Conference on Computer Vision, 1451–1458

  30. Ma T, Latecki LL (2012) Maximum weight cliques with mutex constraints for video object segmentation. IEEE Conference on Computer Vision and Pattern Recognition, 670–677

  31. Mahamud S (2006) Comparing belief propagation and graph cuts for novelty detection. IEEE Conference on Computer Vision and Pattern Recognition, 1154–1159

  32. Matsuyama T, Wada T, Habe H, Tanahashi K (2006) Background subtraction under varying illumination. Syst Comput Japan 37(4):77–88

    Article  Google Scholar 

  33. Ochs P, Brox T (2011) Object segmentation in video: a hierarchical variational approach for turning point trajectories into dense regions. IEEE Conference on Computer Vision

  34. Ochs T, Brox P (2012) Higher order motion models and spectral clustering. IEEE Conference on Computer Vision and Pattern Recognition

  35. Oneata D, Revaud J, Verbeek J, Schmid C (2014) Spatio-temporal object detection proposals. In: European Conference on Computer Vision

  36. Papazoglou A, Ferrari V (2013) Fast object segmentation in unconstrained video. IEEE International Conference on Computer Vision (ICCV), 1777–1784

  37. Price B, Morse B, Cohen S (2009) Livecut: Learning-based interactive video segmentation by evaluation of multiple propagated cues. IEEE Conference on Computer Vision, 779–786

  38. Rahtu E, Kannala J, Salo M, Heikki J (2010) Segmenting salient objects from images and videos. ECCV

  39. Reddy V, Sanderson C, Lovell B (2013) Improved foreground detection via block-based classifier cascade with probabilistic decision integration. IEEE Trans Circuits Syst Video Technol 23(1): 83–93

    Article  Google Scholar 

  40. Rodriguez DM, Ahmed J, Shah M (2011) Action mach: A spatio-temporal maximum average correlation height filter for action recognition. IEEE Conference on Computer Vision and Pattern Recognition

  41. Rubio E, Baena R (2011) Stochastic approximation for background modelling. Int J Comput Vis Image Underst 115(6):735–749

    Article  Google Scholar 

  42. Rubio E, Baena R, Dominguez E (2011) Foreground detection in video sequences with probabilistic self-organizing maps. Int J Neural Syst 21(3):225–246

    Article  Google Scholar 

  43. Sheikh Y, Shah M (2005) Bayesian object detection in dynamic scenes. IEEE Conference on Computer Vision and Pattern Recognition, 74–79

  44. Sun J, Zhang W, Tang X, Shum HY (2006) Background cut. In: Proceedings of ECCV, pp 628–641

  45. Toyama K, Krumm J, Brumitt B, Meyers B (1999) Wallflower: Principles and practice of background maintenance. Seventh International Conference on Computer Vision, 255–261

  46. Tsai D, Flagg M, Rehg J (2010) Motion coherent tracking with multi-label mrf optimization. In: BMVC

  47. Wang Y, Ji Q (2005) A dynamic conditional random field model for object segmentation in image sequences. In: Proceedings of IEEE CVPR, pp 264–270

  48. Xu C, Corso JJ (2012) Evaluation of supervoxel methods for early video processing. IEEE Conference on Computer Vision and Pattern Recognition

  49. Xu X, Huang T (2008) A loopy belief propagation approach for robust background estimation. IEEE Conference on Computer Vision and Pattern Recognition, 1–7

  50. Xu D, Zhang B, Fu H, Lin S (2014) Object-based multiple foreground video co-segmentation. IEEE Conference on Computer Vision and Pattern Recognition

  51. Yin P, Criminisi A, Winn J, Essa I (2007) Tree-based classifiers for bilayer video segmentation. IEEE Conference on Computer Vision and Pattern Recognition, 1–8

  52. Zhang D, Javed O, Shah M (2013) Video object segmentation through spatially accurate and temporally dense extraction of primary object regions. IEEE Conference on Computer Vision and Pattern Recognition, 1451–1458

  53. Zhang G, Jia J, Xiong W, Wong T, Heng P, Bao H (2007) Moving object extraction with a hand-held camera. IEEE Conference on Computer Vision, 1–8

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their valuable comments and suggestions that helped to improve the quality of paper.

Author information

Authors and Affiliations

  1. National University of Computer and Emerging Sciences, H-11/4, Islamabad, Pakistan

    Javid Ullah & Muhammad Arfan Jaffar

  2. Department of Computer Sciences, COMSATS Institue of Information Technology, Abbottabad, Pakistan

    Ahmad Khan

  3. Al Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh, Saudi Arabia

    Muhammad Arfan Jaffar

Authors
  1. Javid Ullah
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Ahmad Khan
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Muhammad Arfan Jaffar
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Muhammad Arfan Jaffar.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ullah, J., Khan, A. & Jaffar, M.A. Motion cues and saliency based unconstrained video segmentation. Multimed Tools Appl 77, 7429–7446 (2018). https://doi.org/10.1007/s11042-017-4655-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4655-4

Keywords