Skip to main content
SpringerLink
Log in

Illumination-aware live videos background replacement using antialiasing optimization

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

We propose a real-time illumination-aware live videos background replacement approach with antialiasing optimization on GPU in this paper. The aim of background replacement for live videos is to substitute the current real-time backgrounds with specially-chosen background images. Here we assume that the camera is stationary and the beginning of the video is only with a pure background scene. We propose the colored locality sensitive histograms (CLSH) considering the influence of other pixels to each pixel in every color channel to improve the performance of background segmentation, which makes the segmentation results robust enough to illumination differences. With the segmentation results, we then introduce a blocked real-time matting approach to enhance the accuracy of the objects’ boundary. Finally, to make the video composition more realistic, we propose a local antialiasing method to recover the distortions on edges. Compared with existing background replacement methods, our approach does not require costly blue/green screen or depth camera, but can produce more reliable video composition results. We have applied hardware GPU parallelism to speed up the live background replacement. Our illumination-aware video background replacement runs very efficiently in real-time, which can be applied for various video applications. The experimental results have shown the efficiency and high-quality rendering of our video background replacement in real-time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Ahmed R, Karmakar G C, Dooley L S (2007) Automatic video background replacement using shape-based probabilistic spatio-temporal object segmentation. In: International conference on information, communications & signal processing

  2. Baf F E, Bouwmans T, Vachon B (2008) Foreground detection using the choquet integral. In: Ninth international workshop on image analysis for multimedia interactive services, pp 187–190

  3. Barnich O, Van D M (2011) Vibe: a universal background subtraction algorithm for video sequences. IEEE Trans Image Process

  4. Bouwmans T (2014) Traditional and recent approaches in background modeling for foreground detection: an overview. Comput Sci Rev

  5. Braham M, Droogenbroeck M V (2016) Deep background subtraction with scene-specific convolutional neural networks. In: International conference on systems, signals and image processing

  6. Brainerd W, Foley T, Kraemer M, Moreton H, Nie Ner M (2016) Efficient gpu rendering of subdivision surfaces using adaptive quadtrees. ACM Trans Graph

  7. Chen M, Wei X, Yang Q, Li Q, Wang G, Yang M H (2017) Spatiotemporal gmm for background subtraction with superpixel hierarchy. IEEE Trans Pattern Anal Mach Intell

  8. Chuang Y Y, Curless B, Salesin D H, Szeliski R (2001) A bayesian approach to digital matting. In: Proceedings of IEEE computer vision and pattern recognition

  9. Evangelio R H, Patzold M, Keller I, Sikora T (2014) Adaptively splitted gmm with feedback improvement for the task of background subtraction. IEEE Trans Inform Forens Secur 9(5):863–874

    Article  Google Scholar 

  10. Fan Q, Zhong F, Lischinski D, Cohen-Or D, Chen B (2015) Jumpcut: non-successive mask transfer and interpolation for video cutout. ACM Trans Graph

  11. Farbman Z, Hoffer G, Lipman Y, Cohen-Or D, Lischinski D (2009) Coordinates for instant image cloning. ACM Trans Graph

  12. Gastal E S L, Oliveira M M (2010) Shared sampling for real-time alpha matting. Comput Graph Forum

  13. He K, Rhemann C, Rother C, Tang X (2011) A global sampling method for alpha matting. In: Computer vision and pattern recognition, pp 2049–2056

  14. He S, Lau R, Yang Q, Wang J (2016) Robust object tracking via locality sensitive histograms. IEEE Trans Circ Syst Vid Technol

  15. Hillman P, Hannah J, Renshaw D (2001) Alpha channel estimation in high resolution images and image sequences. In: Proceedings of IEEE computer vision and pattern recognition

  16. Hofmann M, Tiefenbacher P, Rigoll G (2012) Background segmentation with feedback: the pixel-based adaptive segmenter. In: Computer vision and pattern recognition workshops, pp 38–43

  17. Kaewtrakulpong P, Bowden R (2002) An improved adaptive background mixture model for real-time tracking with shadow detection. Springer, US

  18. Kim W, Jung C (2017) Illumination-invariant background subtraction: comparative review, models, and prospects IEEE Access

  19. Klose F, Wang O, Bazin J C, Magnor M, Sorkine-Hornung A (2015) Sampling based scene-space video processing. ACM Trans Graph

  20. Li B, Sezan M I (2001) Adaptive video background replacement. In: IEEE International conference on multimedia and expo

  21. Lim Y, Park J (2008) Video background replacement using a genetic algorithm. Opt Eng

  22. Lu Y, Bai X, Shapiro L, Wang J (2016) Coherent parametric contours for interactive video object segmentation. In: IEEE Conference on computer vision and pattern recognition, pp 642–650

  23. Ma K L, Painter J S, Hansen C D, Krogh MF (2001) Parallel volume rendering using binary-swap image composition. IEEE Comput Graph Appl

  24. Molnar S, Eyles J, Poulton J (1992) Pixelflow: high-speed rendering using image composition. In: Conference on computer graphics and interactive techniques, SIGGRAPH

  25. Nießner M, Loop C, Meyer M, Derose T (2012) Feature-adaptive GPU rendering of Catmull-Clark subdivision surfaces. ACM Trans Graph 31(1):6:11–6:11

    Article  Google Scholar 

  26. Pérez P, Gangnet M, Blake A (2003) Poisson image editing. ACM Trans Graph 22(3):313–318

    Article  Google Scholar 

  27. Qian R J, Sezan M I (1999) Video background replacement without a blue screen. In: 1999 International conference on image processing, 1999. ICIP 99. Proceedings, vol 4. IEEE, pp 143–146

  28. Rubner Y, Tomasi C, Guibas L J (2000) The earth mover’s distance as a metric for image retrieval. Int J Comput Vis

  29. Ruzon M A, Tomasi C (2000) Alpha estimation in natural images. In: Proceedings of IEEE computer vision and pattern recognition

  30. Sobral A (2013) BGSLibrary: an opencv c++ background subtraction library. In: IX Workshop de Vis?o Computacional (WVC’2013). Rio de Janeiro. https://github.com/andrewssobral/bgslibrary

  31. Sobral A, Bouwmans T (2014) Bgs library: a library framework for algorithms evaluation in foreground/background segmentation. In: Background modeling and foreground detection for video surveillance. CRC Press, Taylor and Francis Group

  32. Sobral A, Vacavant A (2014) A comprehensive review of background subtraction algorithms evaluated with synthetic and real videos. Comput Vis Image Understand 122:4–21

    Article  Google Scholar 

  33. St-Charles P L, Bilodeau G A, Bergevin R (2015) A self-adjusting approach to change detection based on background word consensus. In: IEEE Winter conference on applications of computer vision

  34. Stcharles P L, Bilodeau G A, Bergevin R (2014) Subsense: a universal change detection method with local adaptive sensitivity. IEEE Trans Image Process

  35. Vacavant A, Chateau T, Wilhelm A, Lequivre L (2012) A benchmark dataset for outdoor foreground/background extraction. In: International conference on computer vision, pp 291–300

  36. Vergne R, Barla P, Fleming R W, Granier X (2012) Surface flows for image-based shading design. ACM Trans Graph

  37. Wang J, Agrawala M, Cohen M F (2007) Soft scissors: an interactive tool for realtime high quality matting. In: ACM SIGGRAPH, p 9

  38. Wang L, Gong M, Zhang C, Yang R, Zhang C, Yang Y H (2012) Automatic real-time video matting using time-of-flight camera and multichannel poisson equations. Int J Comput Vis 97(1):104–121

    Article  MATH  Google Scholar 

  39. Wren C R, Azarbayejani A, Darrell T, Pentland A P (1996) Pfinder: real-time tracking of the human body. In: International conference on automatic face and gesture recognition, pp 51–56

  40. Zhang Y, Tang Y L, Cheng K L (2015) Efficient video cutout by paint selection. J Comput Sci Technol 30(3):467–477

    Article  Google Scholar 

  41. Zhang F L, Wu X, Zhang H T, Wang J, Hu S M (2016) Robust background identification for dynamic video editing. Acm Trans Graph 35(6):197

    Google Scholar 

  42. Zhong F, Yang S, Qin X, Lischinski D, Cohen-Or D, Chen B (2014) Slippage-free background replacement for hand-held video. ACM Trans Graph

  43. Zhu Z, Martin R R, Pepperell R, Burleigh A (2016) 3d modeling and motion parallax for improved videoconferencing. Comput Vis Media 2(2):131–142

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank all reviewers for their helpful suggestions and constructive comments, and colleagues for their participating in program testing and helpful discussions. The work is supported by the National Natural Science Foundation of China (No. 61671290), the Key Program for International S&T Cooperation Project (No.2016YFE0129500), UGC grant for research (no. 4055060), NSFC joint projects (No. 61602183, 61379087), and a grant from the Research Grants Council of Hong Kong (No. 28200215).

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

    Qiaoping Hu, Ruimin Shen & Bin Sheng

  2. Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, Hong Kong

    Hanqiu Sun

  3. Faculty of Information Technology, Macau University of Science and Technology, Taipa, Macau

    Ping Li

Authors
  1. Qiaoping Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hanqiu Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ping Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ruimin Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bin Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bin Sheng.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, Q., Sun, H., Li, P. et al. Illumination-aware live videos background replacement using antialiasing optimization. Multimed Tools Appl 77, 24477–24497 (2018). https://doi.org/10.1007/s11042-018-5737-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5737-7

Keywords

Search

Navigation