Skip to main content
SpringerLink
Log in

Variational Image Colorization Models Using Higher-Order Mumford–Shah Regularizers

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

This article introduces variational models for restoring a color image from a grayscale image with color given in only small regions. The models involve the chromaticity color component as in Kang and March (IEEE Trans Image Proc 16(9):2251–2261, 2007), but we make use of higher-order regularization to effectively recover color values of piecewise-smooth images. The first model involves a convex weighted higher-order regularization term, where the weight assists to inhibit the diffusion of chromaticity across the edges. To realize this proposed model, we solve its approximated version obtained by introducing a new variable. We prove the existence of minimizers for both the original and approximated problems, and determine the convergence of their respective solutions. Moreover, we introduce higher-order versions of a Mumford–Shah-like regularizing functional and utilize them for image colorization. The nonconvexity of the proposed functionals enables us to automatically restrain the dispersion of chromaticity across the edges. We also present fast and efficient iterative algorithms for solving the proposed models. Numerical results validate that our models perform more effectively than first-order regularization-based models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Ambrosio, L., Tortorelli, V.M.: Approximation of functionals depending on jumps by elliptic functionals via \(\gamma \)-convergence. Commun. Pure Appl. Math. 43, 999–1036 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  2. Ambrosio, L., Tortorelli, V.M.: On the approximation of free discontinuity problems. Bollettino dellUnione Matematica Italiana B(7)(6), 105–123 (1992)

    MathSciNet  MATH  Google Scholar 

  3. Aubert, G., Kornprobst, P.: Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations, vol. 147, 2nd edn. Applied Mathematical Sciences. Springer (2006)

  4. Bergounioux, M., Piffet, L.: A second-order model for image denoising. Set Valued Anal. Var. Anal. 18(3–4), 277–306 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  5. Besag, J.E.: Digital image processing: towards bayesian image analysis. J. Appl. Stat. 16(3), 395–407 (1989)

    Article  Google Scholar 

  6. Blake, A., Zisserman, A.: Visual Reconstruction. MIT Press, Cambridge (1987)

    Google Scholar 

  7. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2010)

    Article  MATH  Google Scholar 

  8. Bugeau, A., Ta, V.T., Papadakis, N.: Variational exemplar-based image colorization. IEEE Trans. Image Proc. 23(1), 298–307 (2014)

    Article  MathSciNet  Google Scholar 

  9. Candès, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(l_1\) minimization. J. Fourier Anal. Appl. 14(5), 877–905 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  10. Chambolle, A.: An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 20, 89–97 (2004)

    Article  MathSciNet  Google Scholar 

  11. Chan, T.F., Marquina, A., Mulet, P.: High-order total variation-based image restoration. SIAM J. Sci. Comput. 22(2), 503–516 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  12. Charbonnier, P., Féraud, L., Aubert, G., Barlaud, M.: Deterministic edge-preserving regularization in computed imaging. IEEE Trans. Image Proc. 6(2), 298–311 (1997)

    Article  Google Scholar 

  13. Daubechies, I., Devore, R., Fornasier, M., Gntrk, C.S.: Iteratively reweighted least squares minimization for sparse recovery. Commun. Pure Appl. Math. 63(1), 1–38 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  14. Demengel, F.: Fonctions à hessien borné. Ann. Inst. Fourier 34(2), 155–190 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  15. Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two and three space variables. Trans. Am. Math. Soc. 82(3), 421–439 (1956)

    Article  MathSciNet  MATH  Google Scholar 

  16. Esser, E.: Applications of lagrangian-based alternating direction methods and connections to split-bregman. In: UCLA CAM Report 09–31 (2009)

  17. Fang, L., Shen, C., Fan, J., Shen, C.: Image restoration combining a total variational filter and a fourth-order filter. J. Vis. Commun. Image Represent. 18(4), 322–330 (2007)

    Article  Google Scholar 

  18. Fornasier, M.: Nonlinear projection digital image inpainting and restoration methods. J. Math. Imaging Vis. 24(3), 359–373 (2006)

    Article  MathSciNet  Google Scholar 

  19. Geman, D., Reynolds, G.: Constrained restoration and recovery of discontinuities. IEEE Trans. Pattern Anal. Mach. Intell. 14(3), 367–383 (1992)

    Article  Google Scholar 

  20. Geman, D., Yang, C.: Nonlinear image recovery with half-quadratic regularization. IEEE Trans. Image Proc. 4(7), 932–946 (1995)

    Article  Google Scholar 

  21. Geman, S., Geman, D.: Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Trans. Pattern Anal. Mach. Intell. PAMI–6(6), 721–741 (1984)

    Article  MATH  Google Scholar 

  22. Goldstein, T., Osher, S.: The split bregman method for l1-regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  23. Irony, R., Cohen-Or, D., Lischinski, D.: Colorization by example. In: Proceedings of Eurographics Symposium on Rendering, pp. 201–210 (2005)

  24. Jung, M., Kang, M.: Efficient nonsmooth nonconvex optimization for image restoration and segmentation. J. Sci. Comput. 62(2), 336–370 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  25. Kang, S.H., March, R.: Variational models for image colorization via chromaticity and brightness decomposition. IEEE Trans. Image Proc. 16(9), 2251–2261 (2007)

    Article  MathSciNet  Google Scholar 

  26. Lefkimmiatis, S., Bourquard, A., Unser, M.: Hessian-based norm regularization for image restoration with biomedical applications. IEEE Trans. Image Proc. 21(3), 983–995 (2010)

    Article  MathSciNet  Google Scholar 

  27. Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. Proc. SIGGRAPH Conf. 23(3), 689–694 (2004)

    Article  Google Scholar 

  28. Li, S.: Markov Random Field Modeling in Computer Vision, 1st edn. Springer, New York (1995)

    Book  Google Scholar 

  29. Lia, F., Baob, Z., Liuc, R., Zhang, G.: Fast image inpainting and colorization by chambolle’s dual method. J. Vis. Commun. Image Represent. 22(6), 529–542 (2011)

    Article  Google Scholar 

  30. Lysaker, M., Lundervold, A., Tai, X.C.: Noise removal using fourth-order partial differential equation with applications to medical magnetic resonance images in space and time. IEEE Trans. Image Proc. 12(12), 1579–1590 (2003)

    Article  MATH  Google Scholar 

  31. Lysaker, M., Tai, X.C.: Iterative image restoration combining total variation minimization and a second-order functional. Int. J. Comput. Vis. 66(1), 5–18 (2006)

    Article  MATH  Google Scholar 

  32. Mota, J., Xavier, J., Aguiar, P., Puschel, M.: A proof of convergence for the alternating direction method of multipliers applied to polyhedral-constrained functions. arXiv:1112.2295 (2011)

  33. Mumford, D., Shah, J.: Boundary detection by minimizing functionals. In: Proceedings of IEEE International Conference Acoustics, Speech Signal Process. pp. 22–26 (1985)

  34. Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and associated variational problems. Commun. Pure Appl. Math. XLVII, 577–685 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  35. Nikolova, M.: Markovian reconstruction using a GNC approach. IEEE Trans. Image Proc. 8(9), 1204–1220 (1999)

    Article  MathSciNet  Google Scholar 

  36. Nikolova, M., Chan, R.H.: The equivalence of half-quadratic minimization and the gradient linearization iteration. IEEE Trans. Image Proc. 16(6), 1623–1627 (2007)

    Article  MathSciNet  Google Scholar 

  37. Nikolova, M., Ng, M.K., Tam, C.P.: Fast nonconvex nonsmooth minimization methods for image restoration and reconstruction. IEEE Trans. Image Proc. 19(12), 3073–3088 (2010)

    Article  MathSciNet  Google Scholar 

  38. Nikolova, M., Ng, M.K., Zhang, S., Ching, W.: Efficient reconstruction of piecewise constant images using nonsmooth nonconvex minimization. SIAM J. Imaging Sci. 1(1), 2–25 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  39. Ochs, P., Dosovitskiy, A., Brox, T., Pock, T.: An iterated \(\ell _1\) algorithm for non-smooth non-convex optimization in computer vision. IEEE Conference on Computer Vision and Pattern Recognition, 1759–1766 (2013)

  40. Ochs, P., Dosovitskiy, A., Brox, T., Pock, T.: On iteratively reweighted algorithms for nonsmooth nonconvex optimization in computer vision. SIAM J. Imaging Sci. 8(1), 331–372 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  41. Oh, S., Woo, H., Yun, S., Kang, M.: Non-convex hybrid total variation for image denoising. J. Vis. Commun. Image Represent. 24(3), 332–344 (2013)

    Article  Google Scholar 

  42. Papafitsoros, K., Schönlieb, C.B.: A combined first and second order variational approach for image reconstruction. J. Math. Imaging Vis. 48(2), 308–338 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  43. Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12(7), 629–639 (1990)

    Article  Google Scholar 

  44. Pierre, F., Aujol, J.F., Bugeau, A., Papadakis, N., Ta, V.T.: Exemplar-based colorization in rgb color space. In: Proceedings of IEEE International Conference Image Processing, pp. 625–629 (2014)

  45. Pierre, F., Aujol, J.F., Bugeau, A., Papadakis, N., Ta, V.T.: Luminance-chrominance model for image colorization. SIAM J. Imaging Sci. 8(1), 536–563 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  46. Quang, M., Kang, S., Le, T.: Image and video colorization using vector-valued reproducing kernel hilbert spaces. J. Math. Imaging Vis. 37(1), 49–65 (2010)

    Article  MathSciNet  Google Scholar 

  47. Robini, M., Lachal, A., Magnin, I.: A stochastic continuation approach to piecewise constant reconstruction. IEEE Trans. Image Proc. 16(10), 2576–2589 (2007)

    Article  MathSciNet  Google Scholar 

  48. Sapiro, G.: Inpainting the colors. In: Proceedings of IEEE International Conference Image Processing, vol. 2, pp. 698–701 (2005)

  49. Scherzer, O.: Denoising with higher order derivatives of bounded variation and an application to parameter estimation. Computing 60(1), 1–27 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  50. Setzer, S.: Operator splittings, bregman methods and frame shrinkage in image processing. Int. J. Comput. Vis. 92, 265–280 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  51. Setzer, S., Steidl, G.: Variational methods with higher order derivatives in image processing. Approximation 12, 360–386 (2008)

    MathSciNet  MATH  Google Scholar 

  52. Shah, J.: A common framework for curve evolution segmentation and anisotropic diffusion. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, June pp. 136–142 (1996)

  53. Teboul, S., Féraud, L.B., Aubert, G., Barlaud, M.: Variational approach for edge-preserving regularization using coupled PDE’s. IEEE Trans. Image Proc. 7(3), 387–397 (1998)

    Article  Google Scholar 

  54. Vese, L.A., Chan, T.F.: Reduced non-convex functional approximations for image restoration & segmentation. In: UCLA CAM report, pp. 97–56 (1997)

  55. Vogel, C.R., Oman, M.E.: Fast robust total variation based reconstruction of noisy, blurred images. IEEE Trans. Image Proc. 7, 813–824 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  56. Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Proc. 15(5), 1120–1129 (2006)

    Article  Google Scholar 

  57. Yin, W., Osher, S., Goldfarb, D., Darbon, J.: Bregman iterative algorithms for l1-minimization with applications to compressed sensing. SIAM J. Imaging Sci. 1(1), 143–168 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  58. Zhang, T.: Analysis of multi-stage convex relaxation for sparse regularization. J. Mach. Learn. Res. 11, 1081–1107 (2010)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

Miyoun Jung was supported in part by the Hankuk University of Foreign Studies Research Fund and by the Basic Science Research Program through the NRF of Korea funded by the Ministry of Science, ICT, and Future Planning (2013R1A1A3010416). Myungjoo Kang was supported by the Basic Sciences Research Program through the NRF of Korea funded by the Ministry of Science, ICT, and Future Planning (2014R1A2A1A10050531 and 2015R1A5A1009350).

Author information

Authors and Affiliations

  1. Department of Mathematics, Hankuk University of Foreign Studies, Yongin, Korea

    Miyoun Jung

  2. Department of Mathematical Sciences, Seoul National University, Seoul, Korea

    Myungjoo Kang

Authors
  1. Miyoun Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Myungjoo Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miyoun Jung.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jung, M., Kang, M. Variational Image Colorization Models Using Higher-Order Mumford–Shah Regularizers. J Sci Comput 68, 864–888 (2016). https://doi.org/10.1007/s10915-015-0162-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10915-015-0162-9

Keywords

Search

Navigation