Skip to main content
SpringerLink
Log in

Structure–texture image decomposition via non-convex total generalized variation and convolutional sparse coding

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Image decomposition is a fundamental but challenging ill-posed problem in image processing and has been widely applied to compression, enhancement, texture removal, etc. In this paper, we introduce a novel structure–texture image decomposition model via non-convex total generalized variation regularization (NTGV) and convolutional sparse coding (CSC). NTGV aims to characterize the detailed-preserved structural component ameliorating the staircasing artifacts existing in total variation-based models, and CSC aims to characterize image fine-scale textures. Moreover, we incorporate both structure-aware and texture-aware measures to well distinguish structural and textural component. The proposed model is numerically implemented by an alternating minimization scheme based on alternating direction method of multipliers. Experimental results demonstrate the effectiveness of our approach on several applications including texture removal, high dynamic range image tone mapping, detail enhancement and non-photorealistic abstraction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Aach, T., Mota, C., Stuke, I., Muhlich, M., Barth, E.: Analysis of superimposed oriented patterns. IEEE Trans. Image Process. 15(12), 3690–3700 (2006)

    Article  MathSciNet  Google Scholar 

  2. Aujol, J.F., Gilboa, G., Chan, T., Osher, S.: Structure–texture image decomposition modeling, algorithms, and parameter selection. Int. J. Comput. Vis. 67(1), 111–136 (2006)

    Article  MATH  Google Scholar 

  3. Bao, L., Song, Y., Yang, Q., Yuan, H., Wang, G.: Tree filtering: efficient structure-preserving smoothing with a minimum spanning tree. IEEE Trans. Image Process. 23(2), 555–569 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3(3), 492–526 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bristow, H., Eriksson, A., Lucey, S.: Fast convolutional sparse coding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 391–398 (2013)

  6. Buades, A., Coll, B., Morel, J.M.: Nonlocal image and movie denoising. Int. J. Comput. Vision 76(2), 123–139 (2008)

    Article  Google Scholar 

  7. Chalasani, R., Principe, J.C., Ramakrishnan, N.: A fast proximal method for convolutional sparse coding. In: Proceedings of the International Joint Conference on Neural Networks (IJCNN), pp. 1–5. IEEE (2013)

  8. Chen, Q., Xu, J., Koltun, V.: Fast image processing with fully-convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2497–2506 (2017)

  9. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM Rev. 43(1), 129–159 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  10. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893. IEEE (2005)

  11. DeCarlo, D., Santella, A.: Stylization and abstraction of photographs. ACM Trans. Graph. 21(3), 769–776 (2002)

    Article  Google Scholar 

  12. Dowson, N., Salvado, O.: Hashed nonlocal means for rapid image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 485–499 (2010)

    Article  Google Scholar 

  13. Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21(3), 257–266 (2002)

    Article  Google Scholar 

  14. Fan, Q., Yang, J., Hua, G., Chen, B., Wipf, D.: A generic deep architecture for single image reflection removal and image smoothing. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3238–3247 (2017)

  15. Fan, Q., Yang, J., Wipf, D., Chen, B., Tong, X.: Image smoothing via unsupervised learning. ACM Trans. Graph. (TOG) 37(6), 1–14 (2018)

    Article  Google Scholar 

  16. Fan, Y.R., Huang, T.Z., Ma, T.H., Zhao, X.L.: Cartoon-texture image decomposition via non-convex low-rank texture regularization. J. Franklin Inst. 354(7), 3170–3187 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  17. Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph. (TOG) 27(3), 1–10 (2008)

    Article  Google Scholar 

  18. Fattal, R., Lischinski, D., Werman, M.: Gradient domain high dynamic range compression. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques, pp. 249–256 (2002)

  19. Gu, S., Meng, D., Zuo, W., Zhang, L.: Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1708–1716 (2017)

  20. Gu, Y., Yang, X., Gao, Y.: A novel total generalized variation model for image dehazing. J. Math. Imaging Vis. 61(9), 1329–1341 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  21. Guo, X., Li, Y., Ma, J., Ling, H.: Mutually guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 42(3), 694–707 (2018)

    Article  Google Scholar 

  22. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013)

    Article  Google Scholar 

  23. Jähne, B.: Spatio-Temporal Image Processing: Theory and Scientific Applications, vol. 751. Springer, Berlin (1993)

    MATH  Google Scholar 

  24. Jeon, J., Lee, H., Kang, H., Lee, S.: Scale-aware structure-preserving texture filtering. In: Proceedings of the Computer Graphics Forum, vol. 35, pp. 77–86. Wiley, New York (2016)

  25. Kang, M., Kang, M., Jung, M.: Total generalized variation based denoising models for ultrasound images. J. Sci. Comput. 72(1), 172–197 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  26. Kass, M., Solomon, J.: Smoothed local histogram filters. In: ACM SIGGRAPH 2010 Papers, pp. 1–10 (2010)

  27. Kavukcuoglu, K., Sermanet, P., Boureau, Y.L., Gregor, K., Mathieu, M., Cun, Y., et al.: Learning convolutional feature hierarchies for visual recognition. Adv. Neural. Inf. Process. Syst. 23, 1090–1098 (2010)

    Google Scholar 

  28. Kim, Y., Ham, B., Do, M.N., Sohn, K.: Structure–texture image decomposition using deep variational priors. IEEE Trans. Image Process. 28(6), 2692–2704 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  29. Kou, F., Chen, W., Wen, C., Li, Z.: Gradient domain guided image filtering. IEEE Trans. Image Process. 24(11), 4528–4539 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  30. Le, T.M., Vese, L.A.: Image decomposition using total variation and div (bmo). Multiscale Model. Simul. 4(2), 390–423 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  31. Li, Z., Zheng, J., Zhu, Z., Yao, W., Wu, S.: Weighted guided image filtering. IEEE Trans. Image Process. 24(1), 120–129 (2014)

    MathSciNet  MATH  Google Scholar 

  32. Lieu, L.H., Vese, L.A.: Image restoration and decomposition via bounded total variation and negative Hilbert–Sobolev spaces. Appl. Math. Optim. 58(2), 167 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  33. Lindeberg, T.: Scale-space theory: a basic tool for analyzing structures at different scales. J. Appl. Stat. 21(1–2), 225–270 (1994)

    Article  Google Scholar 

  34. Lingling, J., Haiqing, Y., Xiangchu, F.: Adaptive variational models for image decomposition combining staircase reduction and texture extraction. J. Syst. Eng. Electron. 20(2), 254–259 (2009)

    Google Scholar 

  35. Liu, X.: A new Tgv–Gabor model for cartoon-texture image decomposition. IEEE Signal Process. Lett. 25(8), 1221–1225 (2018)

    Article  Google Scholar 

  36. Liu, Z., Li, Y., Wang, W., Liu, L., Chen, R.: Mesh total generalized variation for denoising. IEEE Trans. Visual Comput. Graphics (2021). https://doi.org/10.1109/TVCG.2021.3088118

    Article  Google Scholar 

  37. Lu, C., Song, G.: Image decomposition using adaptive regularization and div (bmo). J. Syst. Eng. Electron. 22(2), 358–364 (2011)

    Article  Google Scholar 

  38. Lu, C., Wang, M.: Alternating direction method for tgv-tgv* based cartoon-texture image decomposition. IET Image Proc. 10(6), 495–504 (2016)

    Article  Google Scholar 

  39. Lu, K., You, S., Barnes, N.: Deep texture and structure aware filtering network for image smoothing. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 217–233 (2018)

  40. Lu, Z., Long, B., Li, K., Lu, F.: Effective guided image filtering for contrast enhancement. IEEE Signal Process. Lett. 25(10), 1585–1589 (2018)

    Article  Google Scholar 

  41. Mairal, J., Elad, M., Sapiro, G.: Sparse representation for color image restoration. IEEE Trans. Image Process. 17(1), 53–69 (2007)

    Article  MathSciNet  Google Scholar 

  42. Meyer, Y.: Oscillating patterns in image processing and nonlinear evolution equations: the fifteenth Dean Jacqueline B. Lewis memorial lectures, vol. 22. American Mathematical Society (2001)

  43. Ochs, P., Dosovitskiy, A., Brox, T., Pock, T.: An iterated \(l^{1}\) algorithm for non-smooth non-convex optimization in computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1759–1766 (2013)

  44. Ochs, P., Dosovitskiy, A., Brox, T., Pock, T.: On iteratively reweighted algorithms for nonsmooth nonconvex optimization in computer vision. SIAM J. Imaging Sci. 8(1), 331–372 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  45. Ono, S.: \(l^{0}\) gradient projection. IEEE Trans. Image Process. 26(4), 1554–1564 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  46. Osher, S., Solé, A., Vese, L.: Image decomposition and restoration using total variation minimization and the \(h^{-1}\) norm. Multiscale Model. Simul. 1(3), 349–370 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  47. Paris, S., Hasinoff, S.W., Kautz, J.: Local Laplacian filters: edge-aware image processing with a Laplacian pyramid. Commun. ACM 58(3), 81–91 (2015)

    Article  Google Scholar 

  48. Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12(7), 629–639 (1990)

    Article  Google Scholar 

  49. Rubinstein, R., Member, S., IEEE, Bruckstein, A.M., Member,: Dictionaries for sparse representation modeling. Proc. IEEE 98(6), 1045–1057 (2010)

  50. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60(1–4), 259–268 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  51. Shu, L., Du, H.: Side window weighted median image filtering. In: Proceedings of the International Conference on Multimedia Systems and Signal Processing, pp. 26–30. Association for Computing Machinery (2020)

  52. Starck, J.L., Elad, M., Donoho, D.L.: Image decomposition via the combination of sparse representations and a variational approach. IEEE Trans. Image Process. 14(10), 1570–1582 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  53. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. 73, 273–282 (1996). https://doi.org/10.1111/j.1467-9868.2011.00771.x

    Article  MathSciNet  MATH  Google Scholar 

  54. Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Proceedings of the International Conference on Computer Vision, pp. 839–846. IEEE (1998)

  55. Valkonen, T., Bredies, K., Knoll, F.: Total generalized variation in diffusion tensor imaging. SIAM J. Imaging Sci. 6(1), 487–525 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  56. Vese, L., Chan, T.F.: Reduced Non-convex Functional Approximations for Image Restoration and Segmentation. University of California, Department of Mathematics, Los Angeles (1997)

    Google Scholar 

  57. Vese, L., Osher, S.: Modeling textures with total variation minimization and oscillating patterns in image processing. J. Sci. Comput. 19(1–3), 553–572 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  58. Wang, C., Zhang, H., Liu, L.: Total generalized variation-based retinex image decomposition. Vis. Comput. 37(1), 77–93 (2021)

    Article  Google Scholar 

  59. Winnemöller, H., Olsen, S.C., Gooch, B.: Real-time video abstraction. ACM Trans. Graph. 25(3), 1221–1226 (2006)

    Article  Google Scholar 

  60. Wohlberg, B.: Efficient convolutional sparse coding. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7173–7177. IEEE (2014)

  61. Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Ma, Y.: Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31(2), 210–227 (2008)

    Article  Google Scholar 

  62. Xu, J., Feng, X., Hao, Y., Han, Y.: Image decomposition and staircase effect reduction based on total generalized variation. J. Syst. Eng. Electron. 25(1), 168–174 (2014)

    Article  Google Scholar 

  63. Xu, J., Feng, X., Hao, Y., Han, Y.: Image decomposition using adaptive second-order total generalized variation. SIViP 8(1), 39–47 (2014)

    Article  Google Scholar 

  64. Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via \(l^{0}\) gradient minimization. In: Proceedings of the SIGGRAPH Asia Conference, pp. 1–12 (2011)

  65. Xu, L., Ren, J., Yan, Q., Liao, R., Jia, J.: Deep edge-aware filters. In: Proceedings of the International Conference on Machine Learning, vol. 37, pp. 1669–1678. PMLR (2015)

  66. Xu, L., Yan, Q., Xia, Y., Jia, J.: Structure extraction from texture via relative total variation. ACM Trans. Graph. (TOG) 31(6), 1–10 (2012)

    Google Scholar 

  67. Xu, P., Wang, W.: Structure-aware window optimization for texture filtering. IEEE Trans. Image Process. 28(9), 4354–4363 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  68. Yang, J., Yu, K., Gong, Y., Huang, T.: Linear spatial pyramid matching using sparse coding for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1794–1801. IEEE (2009)

  69. Yeganeh, H., Wang, Z.: Objective quality assessment of tone-mapped images. IEEE Trans. Image Process. 22(2), 657–667 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  70. Yin, H., Gong, Y., Qiu, G.: Side window filtering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8758–8766 (2019)

  71. Yin, H., Gong, Y., Qiu, G.: Combined window filtering and its applications. Multidimens. Syst. Signal Process. 32(1), 313–333 (2021)

    Article  MATH  Google Scholar 

  72. Yin, W., Goldfarb, D., Osher, S.: Image cartoon-texture decomposition and feature selection using the total variation regularized l\(^{1}\) functional. In: Proceedings of the International Workshop on Variational, Geometric, and Level Set Methods in Computer Vision, pp. 73–84. Springer (2005)

  73. Zeiler, M.D., Krishnan, D., Taylor, G.W., Fergus, R.: Deconvolutional networks. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2528–2535. IEEE (2010)

  74. Zhang, H., Patel, V.M.: Convolutional sparse coding-based image decomposition. In: Proceedings of the British Machine Vision Conference (BMVC) (2016)

  75. Zhang, Q., Shen, X., Xu, L., Jia, J.: Rolling guidance filter. In: Proceedings of the European Conference on Computer Vision, pp. 815–830. Springer (2014)

  76. Zhou, F., Chen, Q., Liu, B., Qiu, G.: Structure and texture-aware image decomposition via training a neural network. IEEE Trans. Image Process. 29, 3458–3473 (2019)

    Article  MATH  Google Scholar 

  77. Zhou, Z., Wang, B., Ma, J.: Scale-aware edge-preserving image filtering via iterative global optimization. IEEE Trans. Multimed. 20(6), 1392–1405 (2017)

    Article  Google Scholar 

Download references

Funding

This study was funded by the Youth Science and Technology Foundation of Gansu (20JR5RA050).

Author information

Authors and Affiliations

  1. Dunhuang Academy, Jiuquan, Gansu, China

    Chunxue Wang

  2. The School of Computer Information Management, Inner Mongolia University of Finance and Economics, Hohhot, Inner Mongolia, China

    Linlin Xu

  3. The School of Mathematical Sciences, University of Science and Technology of China, Hefei, Anhui, China

    Ligang Liu

Authors
  1. Chunxue Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Linlin Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ligang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chunxue Wang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, C., Xu, L. & Liu, L. Structure–texture image decomposition via non-convex total generalized variation and convolutional sparse coding. Vis Comput 39, 1121–1136 (2023). https://doi.org/10.1007/s00371-021-02392-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02392-z

Keywords

Search

Navigation