Skip to main content
SpringerLink
Log in

Painting patches: Reducing flicker in painterly re-rendering of video

  • Research Papers
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

This paper presents a novel method for re-rendering video in a stroke-based painterly style. Previous methods typically place and adjust strokes on a frame by frame basis, guided by an analysis of motion vectors. Our method constructs painting patches which last for multiple frames, and paints them just once, compositing them after placing and clipping each one in each output frame. Painting patches are constructed by clustering pixels with similar motions, representing moving objects. This is done using a multi-frame window, to take account of objects which are present in consecutive frames, and which may occur a few frames apart with occlusion. The appearance of a given cluster across a sequence of frames is warped to a common reference to produce the painting patch; a global optimization of the warp is used to minimize distortion in the painting strokes. This approach outperforms prior algorithms in problem areas of the image, where flickering typically occurs, while producing comparable results elsewhere. In particular, stable strokes are produced at occlusion boundaries where objects emerge, and at image borders exposed by camera panning. A further advantage is consistent rendering of regions before and after brief occlusion, enhancing temporal stability of the output of discontiguous frames.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Hertzmann A. A survey of stroke-based rendering. IEEE Comput Graph Appl, 2003, 23: 70–81

    Article  Google Scholar 

  2. Litwinowicz P C. Processing images and video for an impressionist effect. In: Proceedings of SIGGRAPH’97, Los Angeles, USA, 1997. 407–414

  3. Hertzmann A, Perlin K. Painterly rendering for video and interaction. In: Proceedings of NPAR’00, Annecy, France, 2000. 7–12

  4. Hays J, Essa I. Image and video based painterly animation. In: Proceedings of NPAR’04, Annecy, France, 2004. 113–120

  5. Park Y, Yoon K. Painterly animation using motion maps. Graph Models, 2008, 70: 1–15

    Article  Google Scholar 

  6. Haeberli P. Paint by numbers: abstract image representations. In: Proceedings of SIGGRAPH’90, New York, NY, USA, 1990. 207–214

  7. Hertzmann A. Painterly rendering with curved brush strokes of multiple sizes. In: Proceedings of SIGGRAPH’98, Orlando, USA, 1998. 453–460

  8. Zeng K, Zhao M T, Xiong C M, et al. From image parsing to painterly rendering. ACM Trans Graph, 2009, 29: 2:1–2:11

    Article  Google Scholar 

  9. Kagaya M, Brendel W, Deng Q Q, et al. Video painting with space-time-varying style parameters. IEEE Trans Visual Comput Graph, 2011, 17: 74–87

    Article  Google Scholar 

  10. Baker S, Roth S, Scharstein D, et al. A database and evaluation methodology for optical flow. In: Proceedings of ICCV’07, Rio de Janeiro, Brazil, 2007. 1–31

  11. Bousseau A, Neyret F, Thollot J, et al. Video watercolorization using bidirectional texture advection. ACM Trans Graph, 2007, 26: 104

    Article  Google Scholar 

  12. Zhang S H, Chen T, Zhang Y F, et al. Video-based running water animation in Chinese painting style. Sci China Ser F-Inf Sci, 2009, 52: 162–171

    Article  Google Scholar 

  13. Collomosse J P, Rowntree D, Hall P M. Stroke surfaces: Temporally coherent artistic animations from video. IEEE Trans Vis Comput Graph, 2005, 11: 540–549

    Article  Google Scholar 

  14. Wang J, Xu Y Q, Shum H Y, et al. Video tooning. ACM Trans Graph, 2004, 23: 574–583

    Article  Google Scholar 

  15. Rav-Acha A, Kohli P, Rother C, et al. Unwrap mosaics: a new representation for video editing. ACM Trans Graph, 2008, 27: 1–11

    Article  Google Scholar 

  16. Ayer S, Sawhney H S. Layered representation of motion video using robust maximum-likelihood estimation of mixture models and mdl encoding. In: Proceedings of ICCV’ 95, Washington, DC, USA, 1995. 777–785

  17. Black M J, Jepson A D. Estimating optical flow in segmented images using variable-order parametric models with local deformations. IEEE Trans Patt Anal Mach Intell, 1996, 18: 972–986

    Article  Google Scholar 

  18. Odobez J M, Bouthemy P. Direct incremental model-based image motion segmentation for video analysis. Signal Process, 1998, 66: 143–155

    Article  MATH  Google Scholar 

  19. Xiao J J, Shah M. Motion layer extraction in the presence of occlusion using graph cuts. IEEE Trans Patt Anal Mach Intell, 2005, 27: 1644–1659

    Article  Google Scholar 

  20. Wills J, Agarwal S, Belongie S. A feature-based approach for dense segmentation and estimation of large disparity motion. Int J Comput Vision, 2006, 68: 125–143

    Article  Google Scholar 

  21. Grady L. Random walks for image segmentation. IEEE Trans Patt Anal Mach Intell, 2006, 28: 1768–1783

    Article  Google Scholar 

  22. Bai X, Wang J, Simons D, et al. Video snapcut: robust video object cutout using localized classifiers. In: SIGGRAPH’09: ACM SIGGRAPH 2009, New York, NY, USA, 2009. 1–11

  23. Zach C, Pock T, Bischof H. A duality based approach for realtime tv-l1 optical flow. In: Pattern Recognition (Proc DAGM), Heidelberg, Germany, 2007. 214–223

  24. Wolberg G. Digital Image Warping. New York: John Wiley & Sons, 1990

    Google Scholar 

  25. Zitova B, Flusser J. Image registration methods: a survey. Image Vision Comput, 2003, 21: 977–1000

    Article  Google Scholar 

  26. Rebiere N, Auclair-Fortier M F, Deschenes F. Image mosaicing using local optical flow registration. In: Proceedings of ICPR’08, Tampa, FL, USA, 2008. 1–5

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, Tsinghua University, Beijing, 100084, China

    SongHai Zhang, Qiang Tong & ShiMin Hu

  2. School of Computer Science and Informatics, Cardiff University, Cardiff, Wales, CF24 3AA, UK

    Ralph Martin

Authors
  1. SongHai Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qiang Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. ShiMin Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ralph Martin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to SongHai Zhang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, S., Tong, Q., Hu, S. et al. Painting patches: Reducing flicker in painterly re-rendering of video. Sci. China Inf. Sci. 54, 2592–2601 (2011). https://doi.org/10.1007/s11432-011-4409-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11432-011-4409-2

Keywords

Search

Navigation