Skip to main content
SpringerLink
Log in

Efficient alternating minimization methods for variational edge-weighted colorization models

  • Published:
Advances in Computational Mathematics Aims and scope Submit manuscript

Abstract

Alternating minimization algorithms are developed to solve two variational models, for image colorization based on chromaticity and brightness color system. Image colorization is a task of inpainting color from a small region of given color information. While the brightness is defined on the entire image domain, the chromaticity components are only given on a small subset of image domain. The first model is the edge-weighted total variation (TV) and the second one is the edge-weighted harmonic model that proposed by Kang and March (IEEE Trans. Image Proc. 16(9):2251–2261, 2007). Both models minimize a functional with the unit sphere constraints. The proposed methods are based on operator splitting, augmented Lagrangian, and alternating direction method of multipliers, where the computations can take advantage of multi-dimensional shrinkage and fast Fourier transform under periodic boundary conditions. Convergence analysis of the sequence generated by the proposed methods to a Karush-Kahn-Tucker point and a minimizer of the edge-weighted TV model are established. In several examples, we show the effectiveness of the new methods to colorize gray-level images, where only small patches of colors are given. Moreover, numerical comparisons with quadratic penalty method, augmented Lagrangian method, time marching, and/or accelerated time marching algorithms demonstrate the efficiency of the proposed methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ambrosio, L., Fusco, N., Pallara, D.: Functions of Bounded Variations and Free Discontinuity Problems. Oxford University Press, Oxford (2000)

    MATH  Google Scholar 

  2. Andreani, R., Haeser, G., Schuverdt, M.L., Silva, P.J.S.: A relaxed constant positive linear dependence constraint qualification and applications. Math. Program. 135, 255–273 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bergmann, R., Chan, R.H., Hielscher, R., Persch, J., Steidl, G.: Restoration of manifold-valued images by half-quadratic minimization. Inverse Problems and Imaging 10(2), 281–304 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bergmann, R., Fitschen, J.H., Persch, J., Steidl, G.: Priors with coupled first and second order differences for manifold-valued image processing. arXiv:1709.01343 (2017)

  5. Bergmann, R., Laus, F., Steidl, G., Weinmann, A.: Second order differences of cyclic data and applications in variational denoising. SIAM J. Imag. Sci. 7(4), 2916–2953 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  6. Bergmann, R., Weinmann, A.: A second-order TV-type approach for inpainting and denoising higher dimensional combined cyclic and vector space data. J. Math. Imaging Vision 55(3), 401–427 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  7. Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, New York (1982)

    MATH  Google Scholar 

  8. Bhattacharya, R., Patrangenaru, V.: Large sample theory of intrinsic and extrinsic sample means on manifolds. Ann. Stat. 31(1), 1–29 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  9. Buades, A., Coll, B., Lisani, J.-L., Sbert, C.: Conditional image diffusion. Inverse Problems and Imaging 1(4), 593–608 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  10. Bugeau, A., Ta, V.T., Papadakis, N.: Variational exemplar-based image colorization. IEEE Trans. Image Process. 23, 298–307 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  11. Burns, G.: Museum of broadcast communications: encyclopedia of television. World Wide Web electronic publication (1997)

  12. Liu, X., Chen, C., Li, M., Ye, Y.: On the convergence of multi-block alternating direction method of multipliers and block coordinate descent method. arXiv:1508.00193 (2015)

  13. Cai, X., Han, D., Yuan, X.: The direct extension of ADMM for three-block separable convex minimization models is convergent when one function is strongly convex. Optimization Online (2014)

  14. Carlier, G., Comte, M.: On a weighted total variation minimization problem. J. Funct. Anal. 250, 214–226 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  15. Caselles, V., Coll, B., Morel, J.: Geometry and color in natural images. J. Math. Imaging Vision 16(2), 89–105 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  16. Chan, T., Kang, S.H.: Error analysis for image inpainting. J. Math. Imaging Vision 26, 85–103 (2006)

    Article  MathSciNet  Google Scholar 

  17. Chan, T.F., Kang, S.H., Shen, J.: Total variation denoising and enhancement of color images based on the CB and HSV color models. J. Vis. Commun. Image Represent. 12(4), 422–435 (2001)

    Article  Google Scholar 

  18. Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Math. Program. 64, 81–101 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  19. Chen, W., Ji, H., You, Y.: An augmented Lagrangian method for ℓ 1-regularized optimization problems with orthogonality constraints. SIAM J. Sci. Comput. 38(4), B570–B592 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  20. Chen, Y., Hager, W.W., Yashtini, M., Ye, X., Zhang, H.: Bregman operator splitting with variable stepsize for total variation image reconstruction. Comput. Optim. Appl. 54, 317–342 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  21. Conn, A.R., Gould, N.I.M., Toint, P.L.: A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds. SIAM J. Numerical Analysis 28(2), 545–572 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  22. Conn, A.R., Gould, N.I.M., Toint, P.: A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds. SIAM J. Numer. Anal. 28(2), 545–572 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  23. Cremers, D., Strekalovskiy, E.: Total cyclic variation and generalizations. J. Math. Imaging Vision 47(3), 258–277 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  24. Davis, D., Yin, W.: A three-operator splitting scheme and its optimization applications. Set Valued Anal. Var. Anal. 25(25), 829–858 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  25. Eckstein, J., Silva, P.J.S.: A practical relative error criterion for augmented Lagrangians. Math. Program. 141(1-2), 319–348 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  26. Eckstein, J., Yao, W.: Approximate versions of the alternating direction method of multipliers. Optimization Online (2016)

  27. Fletcher, P.T.: Geodesic regression and the theory of least squares on Riemannian manifolds. Int. J. Comput. Vis. 105(2), 171–185 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  28. Fonseca, I., Leoni, G., Maggi, F., Morini, M.: Exact reconstruction of damaged color images using a total variation model. Annales de l’Institut Henri Poincare (C) Non Linear Analysis 27(5), 1291–1331 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  29. Fornasier, M.: Nonlinear projection digital image inpainting and restoration methods. J. Math. Imaging Vision 24(3), 359–373 (2006)

    Article  MathSciNet  Google Scholar 

  30. Giaquinta, M., Modica, G., Soucek, J.: Variational problems for maps of bounded variation with values in s 1. Calc. Var. 1(1), 87–121 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  31. Giaquinta, M., Mucci, D.: The BV-energy of maps into a manifold: relaxation and density results. Annali della Scuola Normale Superiore di Pisa 5(4), 483–548 (2006)

    MathSciNet  MATH  Google Scholar 

  32. Giaquinta, M., Mucci, D.: Maps of bounded variation with values into a manifold: total variation and relaxed energy. Pure Appl. Math. Q. 3(2), 513–538 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  33. Giusti, E.: Minimal surfaces and functions of bounded variation, volume 80 of monographs in mathematics. Cambridge, MA, Birkhäuser (1984)

  34. Glowinski, R., Pan, T.W., Tai, X.C.: Some facts about operator-splitting and alternating direction methods. UCLA CAM Reports (16-10) (2016)

  35. Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci. 2, 323–343 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  36. Gonzalez, R., Wood, R.: Digital Image Processing. Addison-Wesley, Reading (1992)

    Google Scholar 

  37. Guo, X., Li, F., Ng, M.K.: A fast ℓ1-TV algorithm for image restoration. SIAM J. Sci. Comput. 31(3), 2322–2341 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  38. Hager, W.W.: Minimizing a quadratic over a sphere. SIAM J. Optim. 12(1), 188–208 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  39. Hager, W.W., Ngo, C., Yashtini, M., Zhang, H.: Alternating direction approximate Newton (ADAN) algorithm for ill-conditioned inverse problems with application to Parallel MRI. Journal of Operations Research Society of China 3(2), 139–162 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  40. Hager, W.W., Yashtini, M., Zhang, H.: An O(1/k) convergence rate for the variable stepsize Bregman operator splitting algorithm. SIAM J. Numer. Anal. 53(3), 1535–1556 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  41. Hager, W.W., Zhang, H.: Inexact alternating direction multiplier methods for separable convex optimization. Submitted (2016)

  42. Han, D., Yuan, X.: A note on the alternating direction method of multipliers. J. Optim Theory Appl. 155, 227–238 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  43. He, B., Liao, L., Han, D., Yan, H.: A new inexact alternating directions method for monotone variational inequalities. Math. Program. 92(1), 103–118 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  44. He, B., Tao, M., Yuan, X.: Alternating direction method with Gaussian back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  45. Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)

    Article  MathSciNet  MATH  Google Scholar 

  46. Jung, M., Kang, M.: Variational image colorization models using higher-order Mumford–Shah regularizers. J. Sci. Comput. 86, 864–888 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  47. Kang, S.H., March, R.: Variational models for image colorization via chromaticity and brightness decomposition. IEEE Trans. Image Process. 16(9), 2251–2261 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  48. Kimmel, R., Sochen, N.: Orientation diffusion or how to comb a porcupine. J. Vis. Commun. Image Represent. 13, 238–248 (2001)

    Article  Google Scholar 

  49. Lellmann, Jan, Strekalovskiy, Evgeny, Koetter, Sabrina, Cremers, Daniel: Total variation regularization for functions with values in a manifold. IEEE ICCV, pp. 2944–2951 (2013)

  50. Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. Proceedings of the 2004 SIGGRAPH Conference 23(3), 689–694 (2004)

    Google Scholar 

  51. Li, M., Liao, L., Yuan, X.: Inexact alternating direction methods of multipliers with logarithmic-quadratic proximal regularization. J. Optim Theory Appli. 159, 412–436 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  52. Sun, D., Li, M., Toh, K.C.: A convergent 3-block semi-proximal ADMM for for convex minimization problems with one strongly convex block. arXiv:1410.7933 (2014)

  53. Nesterov, Y.E.: A method for solving the convex programming problem with convergence rate \(\mathcal {O}(1/k^{2})\) (in russian). Dokl. Akad. Nauk SSSR 269, 543–547 (1983)

    MathSciNet  Google Scholar 

  54. Nida, N., Khan, M.U.G.: Efficient colorization of medical imaging based on colour transfer method. In: B Life and Environmental Sciences, vol. 53, pp. 253–261 (2016)

  55. Nocedal, J., Wright, S.J.: Numerical Optimization, vol. 35. Springer, Berlin (1999)

    Book  MATH  Google Scholar 

  56. Papafitsoros, K., Schönlieb, C.B.: A combined first and second order variational approach for image reconstruction. J. Math. Imaging Vision 48(2), 308–338 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  57. Pennec, X.: Intrinsic statistics on Riemannian manifolds: basic tools for geometric measurements. J. Math. Imaging Vis 25, 127–154 (2006)

    Article  MathSciNet  Google Scholar 

  58. Perona, P.: Orientation Diffusion. IEEE Trans. Image Process. 7(3), 457–467 (1998)

    Article  Google Scholar 

  59. Pierre, F., Aujol, J.F., Bugeau, A., Papadakis, N., Ta, V.-T.: Luminance-chrominance model for image colorization. SIAM J. Imag. Sci. 8(1), 536–563 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  60. Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Optimization. Academic Press, New York (1969)

  61. Ha Quang, M., Kang, S.H., Le, T.: Image and video colorization using vector-valued reproducing kernel Hilbert spaces. J. Math. Imaging Vision 37, 49–65 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  62. Rahman, I.U., Drori, I., Stodden, V.C., Donoho, D.L., Schröder, P.: Multiscale representations for manifold-valued data. Multiscale Model Simul. 4(4), 1201–1232 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  63. Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)

    Book  MATH  Google Scholar 

  64. Sapiro, G.: Inpainting the colors. In: ICIP 2005. IEEE International Conference on Image Processing, vol. 2, pp. 698–701 (2005)

  65. Sapiro, G., Ringach, D.: Anisotropic diffusion of multivalued images with applications to color filtering. IEEE Trans. Image Process. 5, 1582–1586 (1996)

    Article  MATH  Google Scholar 

  66. Solodov, M.V., Svaiter, B.F.: A practical relative error criterion for augmented Lagrangians. Math. Oper. Res. 25(2), 214–230 (2000)

    Article  MathSciNet  Google Scholar 

  67. Ma, S., Lin, T., Zhang, S.: On the sublinear convergence rate of multi-block ADMM. arXiv:1408.4265 (2014)

  68. Tai, X.-C., Hahn, J., Chung, G.J.: A fast algorithm for Euler’s elastica model using augmented Lagrangian method. SIAM J. Imag. Sci. 4(1), 313–344 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  69. Tang, B., Sapiro, G., Caselles, V.: Color image enhancement via chromaticity diffusion. IEEE Trans. Image Process. 10, 701–707 (2001)

    Article  MATH  Google Scholar 

  70. Tapia, R.A.: Diagonalized multiplier methods and quasi-Newton methods for constrained optimization. J. Optim. Diagonalized Theory Appl. 22 (1977)

  71. Vese, L., Osher, S.: Numerical methods for p-harmonic flows and applications to image processing. SIAM J. Numer. Anal. 40(6), 2085–2104 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  72. Wang, Y., Yang, J., Yin, W., Zhang, Y.: A new alternating minimization algorithm for total variation image reconstruction. SIAM J Imag Sci 1(3), 248–272 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  73. Weinmann, A: Interpolatory multiscale representation for functions between manifolds. SIAM J. Math. Anal. 44(1), 162–191 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  74. Weinmann, A., Demaret, L., Storath, M.: Total variation regularization for manifold-valued data. SIAM J. Imag. Sci. 7(4), 2226–2257 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  75. Woo, H., Yun, S.: Proximal linearized alternating direction method for multiplicative denoising. SIAM J. Sci Comput. 35(2), B336–B358 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  76. Wu, C., Tai, X.-C.: Augmented Lagrangian method, dual methods, and split Bregman iteration for ROF, vectorial TV, and high order models. SIAM J. Imag. Sci. 3, 300–339 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  77. Xiao, Y.H., Song, H.N.: An inexact alternating directions algorithm for constrained total variation regularized compressive sensing problems. J. Math. Imaging Vision 44(2), 114–127 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  78. Yashtini, M., Hager, W.W., Chen, Y., Ye, X.: Partially parallel MR image reconstruction using sensitivity encoding. In: 2012 IEEE International Conference on Image Processing, pp. 2077–2080. IEEE, Orlando (2012)

  79. Yashtini, M., Kang, S.H.: A fast relaxed normal two split method and an effective weighted TV approach for Euler’s elastica image inpainting. SIAM J. Imag. Sci. 9(4), 1552–1581 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  80. Zhu, W., Tai, X.-C., Chan, T.: Augmented Lagrangian method for a mean curvature based image denoising model. Inverse Problems and Imaging 7, 1409–1432 (2013)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the reviewers for their valuable comments which led the significant improvement of this manuscript.

Funding

This research was supported by the National Science Foundation grant 1344199 and Simons Foundation grant 282311.

Author information

Authors and Affiliations

  1. Department of Mathematics and Statistics, Georgetown University, 327A St. Mary’s Hall, 37th and O Streets, N.W., Washington D.C., 20057, USA

    Maryam Yashtini

  2. School of Mathematics, Georgia Institute of Technology, 686 Cherry Street, Atlanta, GA, 30332-0160, USA

    Sung Ha Kang

  3. Department of Mathematics, University of Alabama, Tuscaloosa, AL, 35486, USA

    Wei Zhu

Authors
  1. Maryam Yashtini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sung Ha Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maryam Yashtini.

Additional information

Communicated by: Raymond H. Chan

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yashtini, M., Kang, S.H. & Zhu, W. Efficient alternating minimization methods for variational edge-weighted colorization models. Adv Comput Math 45, 1735–1767 (2019). https://doi.org/10.1007/s10444-019-09702-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10444-019-09702-z

Keywords

Mathematics Subject Classification (2010)

Search

Navigation