Skip to main content
Springer Nature Link
Log in

Multimodal image enhancement using convolutional sparse coding

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

This paper proposes a wavelet domain-based method for multispectral image super-resolution. The stationary wavelet transform is proposed to decompose the multispectral image into directional wavelet components and for each wavelet component, a joint dictionary learning algorithm is proposed. Using sparse and redundant representations, the proposed approach helps capture intrinsic multispectral features using wavelet domain learning utilizing the up-sampling property of (SWT). The proposed method can learn and recover those image features more accurately. In order to validate the proposed method, we conducted comprehensive experiments. Moreover, we present a comparison of our proposed method with state-of-the-art algorithms over PSNR and SSIM evaluation parameters. The results of the experiments indicate that the proposed method outperforms state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

Data will be available on request.

References

  1. Kaji, S., Kida, S.: Overview of image-to-image translation by use of deep neural networks: denoising, super-resolution, modality conversion, and reconstruction in medical imaging. Radiological physics and technology 12(3), 235–248 (2019)

    Article  Google Scholar 

  2. Han, Q., Yin, Q., Zheng, X., Chen, Z.: Remote sensing image building detection method based on mask r-cnn. Complex & Intelligent Systems 8(3), 1847–1855 (2022)

    Article  Google Scholar 

  3. Lillesand, T., Kiefer, R.W., Chipman, J.: Remote sensing and image interpretation (John Wiley & Sons, 2015)

  4. Du, H., Shi, H., Zeng, D., Zhang, X.P., Mei, T.: The elements of end-to-end deep face recognition: A survey of recent advances. ACM Computing Surveys (CSUR) 54(10s), 1–42 (2022)

    Article  Google Scholar 

  5. Wang, Y., Bashir, S.M.A., Khan, M., Ullah, Q., Wang, R., Song, Y., Guo, Z., Niu, Y.: Remote sensing image super-resolution and object detection: Benchmark and state of the art. Expert Systems with Applications p. 116793 (2022)

  6. Patel, V., Mistree, K.: A review on different image interpolation techniques for image enhancement. International Journal of Emerging Technology and Advanced Engineering 3(12), 129–133 (2013)

    Google Scholar 

  7. Keys, R.: Cubic convolution interpolation for digital image processing. IEEE transactions on acoustics, speech, and signal processing 29(6), 1153–1160 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  8. Gao, F Wang, Y., Yang, Z., Ma, Y., Zhang, Q.: Single image super-resolution based on multi-scale dense attention network. Soft Computing pp. 1–12 (2022)

  9. Dong, C., Loy, C.C., He, K., Tang, X.: in European conference on computer vision (Springer), pp. 184–199(2014)

  10. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence 38(2), 295–307 (2015)

    Article  Google Scholar 

  11. Nazzal, M., Ozkaramanli, H.: Wavelet domain dictionary learning-based single image superresolution. Signal, Image and Video Processing 9(7), 1491–1501 (2015)

    Article  Google Scholar 

  12. Ahmed, J., Baloch, G.L., Ozkaramanli, H.: in 2017 IEEE International Conference on Imaging Systems and Techniques (IST) (IEEE), pp. 1–5 (2017)

  13. Ayas, S., Ekinci, M.: Single image super resolution based on sparse representation using discrete wavelet transform. Multimedia Tools and Applications 77(13), 16685–16698 (2018)

    Article  Google Scholar 

  14. Dai, S., Han, M., Xu, W., Wu, Y., Gong, Y.: in 2007 IEEE Conference on Computer Vision and Pattern Recognition (IEEE), pp. 1–8(2007)

  15. Sun, J., Xu, Z., Shum, H.Y.: in 2008 IEEE Conference on Computer Vision and Pattern Recognition (IEEE), pp. 1–8(2008)

  16. Zhang, X., Wu, X.: Image interpolation by adaptive 2-d autoregressive modeling and soft-decision estimation. IEEE transactions on image processing 17(6), 887–896 (2008)

    Article  MathSciNet  Google Scholar 

  17. Yang, J., Lin, Z., Cohen, S.: in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 1059–1066 (2013)

  18. Hawe, S., Kleinsteuber, M., Diepold, K.: Analysis operator learning and its application to image reconstruction. IEEE Transactions on Image Processing 22(6), 2138–2150 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  19. Mallat, S., Yu, G.: Super-resolution with sparse mixing estimators. IEEE transactions on image processing 19(11), 2889–2900 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  20. Wei, X., Dragotti, P.L.: Fresh-fri-based single-image super-resolution algorithm. IEEE Transactions on Image Processing 25(8), 3723–3735 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  21. Ahmed, J., Baloch, G.L., Ozkaramanli, H.: in 2017 IEEE International Conference on Imaging Systems and Techniques (IST) (IEEE), pp. 1–5 (2017)

  22. Kang, X., Duan, P., Xu, R.: Single image super-resolution based on mapping-vector clustering and nonlinear pixel-reconstruction. Signal Processing: Image Communication 100, 116,501 (2022)

  23. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE transactions on image processing 19(11), 2861–2873 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  24. Candès, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on information theory 52(2), 489–509 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  25. Yang, J., Wang, Z., Lin, Z., Cohen, S., Huang, T.: Coupled dictionary training for image super-resolution. IEEE transactions on image processing 21(8), 3467–3478 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  26. Zeyde, R., Elad, M., Protter, M.: in International conference on curves and surfaces (Springer), pp. 711–730 (2010)

  27. Wang, S., Zhang, L., Liang, Y., Pan, Q.: in 2012 IEEE Conference on Computer Vision and Pattern Recognition (IEEE), pp. 2216–2223 (2012)

  28. Guo, Y., Chen, J., Wang, J., Chen, Q., Cao, J., Deng, Z., Xu, Y., Tan, M.: in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, (Computer Vision Foundation / IEEE, 2020), pp. 5406–5415 (2020)

  29. Marivani, I., Tsiligianni, E., Cornelis, B., Deligiannis, N.: Multimodal deep unfolding for guided image super-resolution. IEEE Transactions on Image Processing 29, 8443–8456 (2020)

    Article  MATH  Google Scholar 

  30. Tao, G., Ji, X., Wang, W., Chen, S., Lin, C., Cao, Y., Lu, T., Luo, D., Tai, Y.: in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, ed. by M. Ranzato, A. Beygelzimer, Y.N. Dauphin, P. Liang, J.W. Vaughan , pp. 22,643–22,654 (2021)

  31. Guo, T., Seyed Mousavi, H., Huu Vu, T., Monga, V.: in Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pp. 104–113 (2017)

  32. Kim, J., Lee, J.K., Lee, K.M.: in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 1646–1654 (2016)

  33. Tomasi, C., Manduchi, R.: in Sixth international conference on computer vision (IEEE Cat. No. 98CH36271) (IEEE), pp. 839–846 (1998)

  34. Kopf, J., Cohen, M.F., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. ACM Transactions on Graphics (ToG) 26(3), 96–es (2007)

  35. He, K., Sun, J., Tang, X.: in European conference on computer vision (Springer), pp. 1–14 (2010)

  36. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE transactions on pattern analysis and machine intelligence 35(6), 1397–1409 (2012)

    Article  Google Scholar 

  37. Shen, X., Yan, Q., Xu, L., Ma, L., Jia, J.: Multispectral joint image restoration via optimizing a scale map. IEEE transactions on pattern analysis and machine intelligence 37(12), 2518–2530 (2015)

    Article  Google Scholar 

  38. Ham, B., Cho, M., Ponce, J.: Robust guided image filtering using nonconvex potentials. IEEE transactions on pattern analysis and machine intelligence 40(1), 192–207 (2017)

    Article  Google Scholar 

  39. Ayas, S., Ekinci, M.: Single image super resolution based on sparse representation using discrete wavelet transform. Multimedia Tools and Applications 77(13), 16685–16698 (2018)

    Article  Google Scholar 

  40. Kumar, N., Verma, R., Sethi, A.: Convolutional neural networks for wavelet domain super resolution. Pattern Recognition Letters 90, 65–71 (2017)

    Article  Google Scholar 

  41. Bae, W., Yoo, J., Chul Ye, J.: in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 145–153 (2017)

  42. Wu, T., Li, W., Jia, S., Dong, Y., Zeng, T.: Deep multi-level wavelet-cnn denoiser prior for restoring blurred image with cauchy noise. IEEE Signal Processing Letters 27, 1635–1639 (2020)

    Article  Google Scholar 

  43. Nazzal, M., Ozkaramanli, H.: Wavelet domain dictionary learning-based single image superresolution. Signal, Image and Video Processing 9(7), 1491–1501 (2015)

    Article  Google Scholar 

  44. Tropp, J.A.: Greed is good: Algorithmic results for sparse approximation. IEEE Transactions on Information theory 50(10), 2231–2242 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  45. Xu, J., Qi, C., Chang, Z.: in 2014 IEEE International Conference on Image Processing (ICIP) (IEEE), pp. 3910–3914 (2014)

  46. Pati, Y.C., Rezaiifar, R., Krishnaprasad, P.S.: in Proceedings of 27th Asilomar conference on signals, systems and computers (IEEE), pp. 40–44 (1993)

  47. Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) 58(1), 267–288 (1996)

    MathSciNet  MATH  Google Scholar 

  48. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM review 43(1), 129–159 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  49. Keys, R.: Cubic convolution interpolation for digital image processing. IEEE transactions on acoustics, speech, and signal processing 29(6), 1153–1160 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  50. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence 38(2), 295–307 (2015)

    Article  Google Scholar 

  51. Zhang, X., Gao, P., Liu, S., Zhao, K., Li, G., Yin, L., Chen, C.W.: Accurate and efficient image super-resolution via global-local adjusting dense network. IEEE Transactions on Multimedia 23, 1924–1937 (2020)

    Article  Google Scholar 

  52. Huang, W., Liao, X., Zhu, L., Wei, M., Wang, Q.: Single-image super-resolution neural network via hybrid multi-scale features. Mathematics 10(4), 653 (2022)

    Article  MathSciNet  Google Scholar 

  53. Song, P., Deng, X., Mota, J.F., Deligiannis, N., Dragotti, P.L., Rodrigues, M.R.: Multimodal image super-resolution via joint sparse representations induced by coupled dictionaries. IEEE Transactions on Computational Imaging 6, 57–72 (2019)

    Article  MathSciNet  Google Scholar 

  54. Yasuma, F., Mitsunaga, T., Iso, D., Nayar, S.K.: Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE transactions on image processing 19(9), 2241–2253 (2010)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Information and Software Engineering, University of Electronic Science and Technology China, Chengdu, China

    Awais Ahmed, She Kun & Abdullah Aman Khan

  2. Department of Electrical Engineering, Sukkur IBA University, Sukkur, Pakistan

    Junaid Ahmed

  3. Department of Computer Science, Air University Multan Campus, Multan, Pakistan

    Shaukat Hayat

  4. Sichuan Artificial Intelligence Research Institute, Yibin, China

    Abdullah Aman Khan

Authors
  1. Awais Ahmed
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. She Kun
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Junaid Ahmed
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Shaukat Hayat
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Abdullah Aman Khan
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

A. Substantial contributions to the conceptual design of the work, Implementations, the acquisition, analysis, or interpretation of data for the work. B. Review the whole paper and include some techniques and suggestions regarding implementation parts. C. Revising it critically for important intellectual content. D. Accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. E. Helped with further experimental evaluation, technical suggestions, revision process, refining, and finalizing the manuscript.

Corresponding author

Correspondence to She Kun.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahmed, A., Kun, S., Ahmed, J. et al. Multimodal image enhancement using convolutional sparse coding. Multimedia Systems 29, 2099–2110 (2023). https://doi.org/10.1007/s00530-023-01074-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-023-01074-1

Keywords