Skip to main content
SpringerLink
Log in

A robust framework for joint background/foreground segmentation of complex video scenes filmed with freely moving camera

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper explores a robust region-based general framework for discriminating between background and foreground objects within a complex video sequence. The proposed framework works under difficult conditions such as dynamic background and nominally moving camera. The originality of this work lies essentially in our use of the semantic information provided by the regions while simultaneously identifying novel objects (foreground) and non-novel ones (background). The information of background regions is exploited to make moving objects detection more efficient, and vice-versa. In fact, an initial panoramic background is modeled using region-based mosaicing in order to be sufficiently robust to noise from lighting effects and shadowing by foreground objects. After the elimination of the camera movement using motion compensation, the resulting panoramic image should essentially contain the background and the ghost-like traces of the moving objects. Then, while comparing the panoramic image of the background with the individual frames, a simple median-based background subtraction permits a rough identification of foreground objects. Joint background-foreground validation, based on region segmentation, is then used for a further examination of individual foreground pixels intended to eliminate false positives and to localize shadow effects. Thus, we first obtain a foreground mask from a slow-adapting algorithm, and then validate foreground pixels (moving visual objects + shadows) by a simple moving object model built by using both background and foreground regions. The tests realized on various well-known challenging real videos (across a variety of domains) show clearly the robustness of the suggested solution. This solution, which is relatively computationally inexpensive, can be used under difficult conditions such as dynamic background, nominally moving camera and shadows. In addition to the visual evaluation, spatial-based evaluation statistics, given hand-labeled ground truth, has been used as a performance measure of moving visual objects detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Alzoubia H, Pan WD (2008) Fast and accurate global motion estimation algorithm using pixel subsampling. Inf Sci 178(17):3415–3425

    Article  Google Scholar 

  2. Amri S, Zagrouba E, Barhoumi W (2008) Background construction for video sequences with complex motions. In: Proceedings of the international Group of e-Systems Research and Applications, Hammamet, Tunisia, pp. 11–26

  3. Barhoumi W, Zagrouba E, Solaiman B, Ghorbel F (2003) Fusion de l’information par la théorie de l’évidence : Application en diagnostic du mélanome. In: Proceedings of Sciences Electroniques, Technologies de l’Information et des Télécommunications, Sousse, Tunisia

  4. Benedek C, Sziranyi T (2006) Markovian framework for foreground-background-shadow separation of real world video scenes. In: Proceedings of the Asian Conference on Computer Vision, Hyderabad, India, pp. 898–907

  5. Bugeau A, Perez P (2007) Detection and segmentation of moving objects in highly dynamic scenes. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, USA, pp: 1–8

  6. Bunyak F, Ersoy I, Subramanya SR (2005) Shadow detection by combined photometric invariants for improved foreground segmentation. In: Proceedings of the Seventh IEEE Workshop on Application of Computer Vision, Breckenridge, USA, pp. 510–515

  7. Cheung SS, Kamath C (2004) Robust techniques for background subtraction in urban traffic video. In: Proceedings of International Conference on Visual Communications and Image Processing, San Jose, USA, pp. 881–892

  8. Cohen I, Medioni G (1999) Detecting and tracking moving objects for video surveillance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Fort Collins, USA, pp. 318–325

  9. Colombari A, Fusiello A, Murino V (2007) Segmentation and tracking of multiple video objects. Pattern Recogn 40(1):1307–1317

    Article  MATH  Google Scholar 

  10. Colombari A, Fusiello A, Murino V (2007) Video objects segmentation by robust background modeling. In: Proceedings of the IEEE International Conference on Image Analysis and Processing, Modena, Italy, pp. 155–164

  11. Criminisi A, Cross G, Blake A, Kolmogorov V (2006) Bilayer segmentation of live video. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New York, USA, pp. 53–60

  12. Cucchiara R, Grana C, Piccardi M, Prati A (2003) Detecting moving objects, ghosts, and shadows in video streams. IEEE Trans Pattern Anal Mach Intell 25(10):1–6

    Article  Google Scholar 

  13. Dahyot R (2006) Unsupervised camera motion estimation and moving object detection in videos. In: Proceedings of the Irish Machine Vision and Image Processing Conference, Dublin, Ireland, pp. 102–109

  14. De Beecka KO, Yu-Hua Gu I, Liyuan L, Vibergb M, De Moor B (2006) Region-based statistical background modeling for foreground object segmentation. In: Proceedings of the IEEE International Conference on Image Processing, Atlanta, USA, pp. 3317–3320

  15. Eledath J, McDowell L, Hansen M, Wixson L, Pope A, Gendel G (1998) Real-time fixation, mosaic construction and moving object detection from a moving camera. In: Proceedings of the Fourth IEEE Workshop on Applications of Computer Vision, Princeton, USA

  16. Elhabian SY, El-Sayed KM, Ahmed SH (2008) Moving object detection in spatial domain using background removal techniques—State-of-art. Recent Patents on Computer Science, Bentham Science Publishers Ltd, 1(1), pp. 32–54

  17. Farin D, De With HN, Effelsberg W (2004) Video-object segmentation using multi-sprite background subtraction. In: Proceedings of the IEEE International Conference on Multimedia and Expo, Taipei, Taiwan

  18. Farin D, De With PH, Effelsberg W (2003) Robust background estimation for complex video sequences. In: Proceedings of the IEEE International Conference on Image Processing, pp. 145–148

  19. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Comm. ACM 24(6):381–395

    Google Scholar 

  20. Gabriel PF, Verly JG, Piater JH, Genon A (2003) The state of the art in multiple object tracking under occlusion in video sequences. In: Proceedings of the Advanced Concepts for Intelligent Vision Systems, Ghent, Belgium

  21. Gracias NRE, Gleason ACR, Negahdaripour S, Mahoor MH (2006) Fast image blending using watersheds and graph cuts. In: Proceedings of the British Machine Vision Conference , Edinburgh, UK

  22. Gouze A, De Roovern C, Macq B, Herbulot A, Debreuve E, Barlaud M (2005) Watershed-driven active contours for moving object segmentation. In: Proceedings of IEEE International Conference on Image Processing, Genoa, Italy

  23. Hartley R, Zisserman A (2004) Multiple view geometry in computer vision. Cambridge University Press, Second edition

  24. Harville M, Gordon G, Woodfill J (2001) Foreground segmentation using adaptive mixture models in color and depth. In: Proceedings of the IEEE Workshop on Detection and Recognition of Events in Video, Vancouver, Canada, pp. 3–11

  25. Hsu CT, Tsan YC (2004) Mosaics of video sequences with moving objects. Signal Process Image Comm 19(1):81–98

    Article  Google Scholar 

  26. Ibanez L, Schroeder W, Ng L, Cates J (2003) The ITK software guide: The insight segmentation and registration toolkit, Kitware Inc.Publisher

  27. Jehan-Besson S, Barlaud M, Aubert G (2001) Region-based active contours for video object segmentation with camera compensation. In: Proceedings of the IEEE International Conference on Image Processing, Thessaloniki, Greece, pp. 61–64

  28. Jung YK, Lee KW, Woo DM, Ho YS (2004) Automatic video object tracking using a mosaic-based background. In: Proceedings of the Pacific Rim Conference on Multimedia, Tokyo, Japan, pp. 866–873

  29. Kentaro T, John K, Barry B, Brian M (1999) Wallflower: Principles and practice of background maintenance. In: Proceedings of the Seventh International Conference on Computer Vision, Kerkyra, Greece, pp. 255–261

  30. Kim N, Kim I, Kim H (2006) Video surveillance using dynamic configuration of multiple active cameras. In: Proceedings of the IEEE International Conference on Image Processing, Atlanta, USA, pp. 1761–1764

  31. Kolmogorov V, Criminisi A, Blake A, Cross G, Rother C (2005) Bi-layer segmentation of binocular stereo video. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, USA, pp. 407–414

  32. Lafon D, Ramananantoandro T (2002) Color images. Image Anal Stereol 21(1):61–74

    Google Scholar 

  33. Lu L, Hager G (2006) Dynamic foreground-background extraction from images and videos using random patches. In: Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, Canada, pp. 929–936

  34. Lu Y, Gao W, Wu F (2002) Automatic video segmentation using a novel background model. In: Proceedings of IEEE International Symposium on Circuits and Systems, Phoenix-Scottsdale, USA, pp. 807–810

  35. Mann S (2002) VideoOrbits: The projective geometry renaissance. Intelligent Image Processing, John Wiley & Sons, pp. 233–294

  36. Mittal A, Monnet A, Paragios N (2009) Scene modeling and change detection in dynamic scenes: A subspace approach. Comput Vis Image Underst 113(1):63–79

    Article  Google Scholar 

  37. Paragios N, Tziritas G (1999) Adaptive detection and localization of moving objects in image sequences. Signal Process Image Comm 14:277–296

    Article  Google Scholar 

  38. Patwardhan KA, Sapiro G, Morellas V (2008) Robust foreground detection in video using pixel layers. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(4):746–751

    Article  Google Scholar 

  39. Rao NI, Di H, Xu GY (2007) Panoramic background model under free moving camera. In: Proceedings of the Fourth International Conference on Fuzzy Systems and Knowledge Discovery, pp. 639–643

  40. Ren Y, Chua CS, Ho YK (2003) Statistical background modeling for non-stationary camera. Pattern Recognit Lett 24(1):183–196

    Article  MATH  Google Scholar 

  41. Rosin P, Ioannidis E (2003) Evaluation of global image thresholding for change detection. Pattern Recognit Lett 24(14):2345–2356

    Article  MATH  Google Scholar 

  42. Rymel J, Renno J, Greenhill D, Orwell J, Jones GA (2004) Adaptive eigen-backgrounds for object detection. In: Proceedings of the International Conference on Image Processing, Singapore, pp. 1847–1850

  43. Sarkar S, Majchrzak D, Korimilli K (2002) Perceptual organization based computational model for robust segmentation of moving objects. Comput Vis Image Underst 86(3):141–170

    Article  MATH  Google Scholar 

  44. Sawhney HS, Ayer S (1996) Compact representations of videos through dominant and multiple motion estimation. IEEE Trans Pattern Anal Mach Intell 18(8):814–830

    Article  Google Scholar 

  45. Shao J, Zhou SK, Chellappa R (2004) Simultaneous background and foreground modeling for tracking in surveillance video. In: Proceedings of the International Conference on Image Processing, Singapore, pp. 1053–1056

  46. Shih MY, Chang YJ, Fu BC, Huang CC (2007) Motion-based background modeling for moving object detection on moving platforms. In: Proceedings of International Conference on Computer Communications and Networks, Hawaii, USA, pp. 1178–1182

  47. Spagnolo P, Orazio TD, Leo M, Distante A (2006) Moving object segmentation by background subtraction and temporal analysis. Image Vis Comput 24(5):413–423

    Article  Google Scholar 

  48. Sugaya Y, Kanatani K (2004) Extracting moving objects from a moving camera video sequence. In: Proceedings of the Tenth Symposium on Sensing via Image Information, Yokohama, Japan, pp. 279–284

  49. Talukder A, Matthies L (2004) Real-time detection of moving objects from moving vehicles using dense stereo and optical flow. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3718–3725

  50. Tavakkoli A, Nicolescu M, Bebis G (2008) Approach for background modeling in videos with quasi-stationary backgrounds. International Journal of Artificial Intelligence Tools 17(4):635–658

    Article  Google Scholar 

  51. Thakoor N, Gao J (2005) Automatic extraction and localization of multiple moving objects with stereo camera in motion. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics, Hawaii, USA, pp. 1269–1274

  52. Vincent L, Soille P (1991) Watersheds in digital spaces: an efficient algorithm based on immersion simulation. IEEE Trans Pattern Anal Mach Intell 13(6):563–598

    Article  Google Scholar 

  53. Yan WQ, Wang J, Kankanhalli MS (2005) Automatic video logo detection and removal. Multimedia Systems 10(5):379–391

    Article  Google Scholar 

  54. Yilmaz A, Shah M (2004) Contour-based object tracking with occlusion handling in video acquired using mobile cameras. IEEE Trans Pattern Anal Mach Intell 26(11):1531–1536

    Article  Google Scholar 

  55. Yu Q, Médioni G (2007) Map-enhanced detection and tracking from a moving platform with local and global association. In: Proceedings of the IEEE Workshop on Motion and Video Computing, Austin, USA

  56. Yu T, Zhang C, Cohen M, Rui Y, Wu Y (2007) Monocular video foreground/background segmentation by tracking spatial-color gaussian mixture models. In: Proceedings of the IEEE Workshop on Motion and Video Computing, Austin, USA

  57. Yu Y, Mann G, Gosine R (2007) Task-driven moving object detection for robots using visual attention. In: Proceedings of the IEEE-RAS International Conference on Humanoid Robots, Pittsburgh, USA

  58. Zagrouba E, Barhoumi W, Amri S (2009) An efficient image mosaicing method based on multifeature matching. Mach Vision Appl 20(3):139–162

    Google Scholar 

  59. Zhang Y, Kiselewich SJ, Bauson WA, Hammoud R (2006) Robust moving object detection at distance in the visible spectrum and beyond using a moving camera. In: Proceedings of the Workshop OTCVBS, Conference on Computer Vision and Pattern Recognition, pp. 131–131

  60. Zhao W (2006) Flexible image blending for image mosaicing with reduced artifacts. Pattern Recognit Artif Intell 20(4):609–628

    Google Scholar 

  61. Zhong J, Sclaroff S (2003) Segmenting foreground objects from a dynamic textured background via a robust kalman filter. In: Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France

  62. Zhu J, Schwartz SC, Liu B (2005) A transform domain approach to real-time foreground segmentation in video sequences. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Philadelphia, USA, pp. 685–688

Download references

Author information

Authors and Affiliations

  1. Equipe de Recherche Systèmes Intelligents en Imagerie et Vision Artificielle (SIIVA) Institut Supérieur d’Informatique, 2 Rue Abou Rayhane El Bayrouni, 2080, Ariana, Tunisia

    Slim Amri, Walid Barhoumi & Ezzeddine Zagrouba

Authors
  1. Slim Amri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Walid Barhoumi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ezzeddine Zagrouba
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Walid Barhoumi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Amri, S., Barhoumi, W. & Zagrouba, E. A robust framework for joint background/foreground segmentation of complex video scenes filmed with freely moving camera. Multimed Tools Appl 46, 175–205 (2010). https://doi.org/10.1007/s11042-009-0348-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-009-0348-y

Keywords

Search

Navigation