Skip to main content
Springer Nature Link
Log in

Inpainting-Based Video Compression in FullHD

  • Conference paper
  • First Online:
Scale Space and Variational Methods in Computer Vision (SSVM 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12679))

Abstract

Compression methods based on inpainting are an evolving alternative to classical transform-based codecs for still images. Attempts to apply these ideas to video compression are rare, since reaching real-time performance is very challenging. Therefore, current approaches focus on simplified frame-by-frame reconstructions that ignore temporal redundancies. As a remedy, we propose a highly efficient, real-time capable prediction and correction approach that fully relies on partial differential equations (PDEs) in all steps of the codec: Dense variational optic flow fields yield accurate motion-compensated predictions, while homogeneous diffusion inpainting is applied for intra prediction. To compress residuals, we introduce a new highly efficient block-based variant of pseudodifferential inpainting. Our novel architecture outperforms other inpainting-based video codecs in terms of both quality and speed. For the first time in inpainting-based video compression, we can decompress FullHD (1080p) videos in real-time with a fully CPU-based implementation, outperforming previous approaches by roughly one order of magnitude.

This work has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 741215, ERC Advanced Grant INCOVID). We thank Matthias Augustin for sharing his in-depth knowledge in pseudodifferential inpainting.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Andris, S., Peter, P., Weickert, W.: A proof-of-concept framework for PDE-based video compression. In: Proceedings of the 2016 Picture Coding Symposium. IEEE Computer Society Press, Nürnberg, Germany (2016)

    Google Scholar 

  2. Arai, Y., Agui, T., Nakajima, M.: A fast DCT-SQ scheme for images. IEICE Trans. 71(11), 1095–1097 (1988)

    Google Scholar 

  3. Augustin, M., Weickert, J., Andris, S.: Pseudodifferential inpainting: the missing link between PDE- and RBF-based interpolation. In: Lellmann, J., Burger, M., Modersitzki, J. (eds.) SSVM 2019. LNCS, vol. 11603, pp. 67–78. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22368-7_6

    Chapter  Google Scholar 

  4. Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comput. Vision 92(1), 1–31 (2011)

    Article  Google Scholar 

  5. Breuß, M., Hoeltgen, L., Radow, G.: Towards PDE-based video compression with optimal masks prolongated by optic flow. J. Math. Imag. Vision 62, 1–13 (2020)

    Google Scholar 

  6. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_3

    Chapter  Google Scholar 

  7. Bull, D.: Communicating Pictures: A Course in Image and Video Coding. Academic Press, Cambridge, MA (2014)

    Book  Google Scholar 

  8. Carlsson, S.: Sketch based coding of grey level images. Signal Process. 15(1), 57–83 (1988)

    Article  Google Scholar 

  9. Chen, W., Mied, R.: Optical flow estimation for motion-compensated compression. Image Vision Comput. 31(3), 275–289 (2013)

    Article  Google Scholar 

  10. Collet, Y.: Finite state entropy (2013). https://github.com/Cyan4973/FiniteStateEntropy

  11. Deuflhard, P.: Cascadic conjugate gradient methods for elliptic partial differential equations: algorithm and numerical results. In: Keyes, D.E., Xu, J. (eds.) Contemporary Mathematics, vol. 180, pp. 29–29. American Mathematical Society, Procidence, RI (1994)

    Google Scholar 

  12. Doshkov, D., Ndjiki-Nya, P., Lakshman, H., Koppel, M., Wiegand, T.: Towards efficient intra prediction based on image inpainting methods. In: Proceedings of the 27th Picture Coding Symposium, pp. 470–473. IEEE Computer Society Press, Nagoya, Japan (2010)

    Google Scholar 

  13. Galić, I., Weickert, J., Welk, M., Bruhn, A., Belyaev, A., Seidel, H.P.: Image compression with anisotropic diffusion. J. Math. Imag. Vision 31(2–3), 255–269 (2008)

    Article  MathSciNet  Google Scholar 

  14. Han, S.C., Podilchuk, C.: Video compression with dense motion fields. IEEE Trans. Image Process. 10(11), 1605–1612 (2001)

    Article  Google Scholar 

  15. Haskell, B.G., Puri, A., Netravali, A.N.: Digital Video: An Introduction to MPEG-2. Springer, Berlin (1996)

    Google Scholar 

  16. Hoffmann, S., Plonka, G., Weickert, J.: Discrete Green’s functions for harmonic and biharmonic inpainting with sparse atoms. In: Tai, X.-C., Bae, E., Chan, T.F., Lysaker, M. (eds.) EMMCVPR 2015. LNCS, vol. 8932, pp. 169–182. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-14612-6_13

    Chapter  Google Scholar 

  17. Horn, B., Schunck, B.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)

    Article  Google Scholar 

  18. Jost, F., Peter, P., Weickert, J.: Compressing flow fields with edge-aware homogeneous diffusion inpainting. In: Proceedings of the 45th International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 2198–2202. IEEE Computer Society Press, Barcelona, Spain (2020)

    Google Scholar 

  19. Köstler, H., Stürmer, M., Freundl, C., Rüde, U.: PDE based video compression in real time. Tech. Rep. 07–11, Lehrstuhl für Informatik 10, University Erlangen-Nürnberg, Germany (2007)

    Google Scholar 

  20. Li, B., Han, J., Xu, Y.: Co-located reference frame interpolation using optical flow estimation for video compression. In: Proceedings of the 2018 Data Compression Conference, pp. 13–22. IEEE Computer Society Press, Snowbird, UT (2018)

    Google Scholar 

  21. Liu, D., Sun, X., Wu, F., Zhang, Y.Q.: Edge-oriented uniform intra prediction. IEEE Trans. Image Process. 17(10), 1827–1836 (2008)

    Article  MathSciNet  Google Scholar 

  22. Ottaviano, G., Kohli, P.: Compressible motion fields. In: Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2251–2258. IEEE Computer Society Press, Oregon, OH (2013)

    Google Scholar 

  23. Peter, P., Kaufhold, L., Weickert, J.: Turning diffusion-based image colorization into efficient color compression. IEEE Trans. Image Process. 26(2), 860–869 (2016)

    Article  MathSciNet  Google Scholar 

  24. Peter, P., Schmaltz, C., Mach, N., Mainberger, M., Weickert, J.: Beyond pure quality: progressive modes, region of interest coding, and real time video decoding for PDE-based image compression. J. Vis. Commun. Image Represent. 31(4), 253–265 (2015)

    Article  Google Scholar 

  25. Rissanen, J.J.: Generalized Kraft inequality and arithmetic coding. IBM J. Res. Dev. 20(3), 198–203 (1976)

    Article  MathSciNet  Google Scholar 

  26. Roosendaal, T.: Sintel. In: ACM SIGGRAPH 2011 Computer Animation Festival, p. 71. New York, NY, USA (2011)

    Google Scholar 

  27. Sanchez, V., Garcia, P., Peinado, A.M., Segura, J.C., Rubio, A.J.: Diagonalizing properties of the discrete cosine transforms. IEEE Trans. Signal Process. 43(11), 2631–2641 (1995)

    Article  Google Scholar 

  28. Schmaltz, C., Peter, P., Mainberger, M., Ebel, F., Weickert, J., Bruhn, A.: Understanding, optimising, and extending data compression with anisotropic diffusion. Int. J. Comput. Vision 108(3), 222–240 (2014)

    Article  MathSciNet  Google Scholar 

  29. Schmaltz, C., Weickert, J.: Video compression with 3-D pose tracking, PDE-based image coding, and electrostatic halftoning. In: Pinz, A., Pock, T., Bischof, H., Leberl, F. (eds.) DAGM/OAGM 2012. LNCS, vol. 7476, pp. 438–447. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-32717-9_44

    Chapter  Google Scholar 

  30. Strang, G., MacNamara, S.: Functions of difference matrices are Toeplitz plus Hankel. SIAM Rev. 56(3), 525–546 (2014)

    Article  MathSciNet  Google Scholar 

  31. Tan, T.K., Boon, C.S., Suzuki, Y.: Intra prediction by template matching. In: Proceedings of the 2006 IEEE International Conference on Image Processing, pp. 1693–1696. IEEE Computer Society Press, Atlanta, GA, USA (2006)

    Google Scholar 

  32. Taubman, D.S., Marcellin, M.W. (eds.): JPEG 2000: Image Compression Fundamentals. Standards and Practice. Kluwer, Boston (2002)

    Google Scholar 

  33. Wu, C.-Y., Singhal, N., Krähenbühl, P.: Video compression through image interpolation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 425–440. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01237-3_26

    Chapter  Google Scholar 

  34. Zhang, Y., Lin, Y.: Improving HEVC intra prediction with PDE-based inpainting. In: Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), IEEE Computer Society Press, Chiang Mai, Thailand (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Mathematical Image Analysis Group, Faculty of Mathematics and Computer Science, Saarland University, Campus E1.7, 66041, Saarbrücken, Germany

    Sarah Andris, Pascal Peter, Rahul Mohideen Kaja Mohideen, Joachim Weickert & Sebastian Hoffmann

Authors
  1. Sarah Andris
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pascal Peter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rahul Mohideen Kaja Mohideen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joachim Weickert
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sebastian Hoffmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Sarah Andris , Pascal Peter , Rahul Mohideen Kaja Mohideen , Joachim Weickert or Sebastian Hoffmann .

Editor information

Editors and Affiliations

  1. UNICAEN, GREYC – Normandy University, Caen, France

    Abderrahim Elmoataz

  2. ENSICAEN, GREYC – Normandy University, Caen, France

    Jalal Fadili

  3. CNRS, GREYC – Normandy University, Caen, France

    Yvain Quéau

  4. UNICAEN, GREYC – Normandy University, Caen, France

    Julien Rabin

  5. ENSICAEN, GREYC – Normandy University, Caen, France

    Loïc Simon

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Andris, S., Peter, P., Mohideen Kaja Mohideen, R., Weickert, J., Hoffmann, S. (2021). Inpainting-Based Video Compression in FullHD. In: Elmoataz, A., Fadili, J., Quéau, Y., Rabin, J., Simon, L. (eds) Scale Space and Variational Methods in Computer Vision. SSVM 2021. Lecture Notes in Computer Science(), vol 12679. Springer, Cham. https://doi.org/10.1007/978-3-030-75549-2_34

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-75549-2_34

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-75548-5

  • Online ISBN: 978-3-030-75549-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics