Skip to main content
SpringerLink
Log in

High-to-low-level feature matching and complementary information fusion for reference-based image super-resolution

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The aim of the reference-based image super-resolution (RefSR) is to reconstruct high-resolution (HR) when a reference (Ref) image with similar content as that of the low-resolution (LR) input is given. In the task, the quality of existing approaches degrades severely when there are several similar objects but different contents. Besides, not all similar information in the reference image is useful for the input image. Therefore, we propose high-to-low-level feature matching and complementary information fusion (HMCF) network for RefSR. The matching strategy adopts high-level to low-level feature matching to distinguish similar objects but different contents according to high-level semantics. The complementary information fusion module utilizes the channel and spatial attention to select the complement information for LR image and keeps the pixel consistency of input and Ref image. We perform extensive experiments to demonstrate that our proposed HMCF obtains the SOTA performance on the RefSR benchmarks and presents a high visual quality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data Availability Statement

The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Mahapatra, D., Bozorgtabar, B., Garnavi, R.: Image super-resolution using progressive generative adversarial networks for medical image analysis. Comput. Med. Imaging Graph. 71, 30–39 (2019)

    Article  Google Scholar 

  2. Qiu, D., Zheng, L., Zhu, J., Huang, D.: Multiple improved residual networks for medical image super-resolution. Futur. Gener. Comput. Syst. 116, 200–208 (2021)

    Article  Google Scholar 

  3. Demirel, H., Anbarjafari, G.: Discrete wavelet transform-based satellite image resolution enhancement. IEEE Trans. Geosci. Remote Sens. 49(6), 1997–2004 (2011)

    Article  Google Scholar 

  4. Xia, R., Chen, Y., Ren, B.: Improved anti-occlusion object tracking algorithm using unscented Rauch-Tung-Striebel smoother and Kernel correlation filter, J. King Saud Univ-Comput. Inf. Sci.

  5. Zhang, J., Feng, W., Yuan, T., Wang, J., Sangaiah, A.K.: Scstcf: spatial-channel selection and temporal regularized correlation filters for visual tracking. Appl. Soft Comput. 118, 108485 (2022)

    Article  Google Scholar 

  6. Chen, Y., Liu, L., Phonevilay, V., Gu, K., Xia, R., Xie, J., Zhang, Q., Yang, K.: Image super-resolution reconstruction based on feature map attention mechanism. Appl. Intell. 51(7), 4367–4380 (2021)

    Article  Google Scholar 

  7. Zheng, H., Ji, M., Wang, H., Liu, Y., Fang, L.: Crossnet: An end-to-end reference-based super resolution network using cross-scale warping. In: Proceedings of the European conference on computer vision (ECCV) pp. 88–104 (2018)

  8. Zhang, Z., Wang, Z., Lin, Z., Qi, H.: Image super-resolution by neural texture transfer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7982–7991 (2019)

  9. Yang, F., Yang, H., Fu, J., Lu, H., Guo, B.: Learning texture transformer network for image super-resolution. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR) pp. 5791–5800 (2020)

  10. Lu, L., Li, W., Tao, X., Lu, J., Jia, J.: Masa-sr: Matching acceleration and spatial adaptation for reference-based image super-resolution. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR) pp. 6368–6377 (2021)

  11. Jiang, Y., Chan, K., Wang, X., Loy, C. C., Liu, Z.: Robust reference-based super-resolution via c2-matching. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR) pp. 2103–2112 (2021)

  12. Xia, B., Tian, Y., Hang, Y., Yang, W., Zhou, J.: Coarse-to-fine embedded patchmatch and multi-scale dynamic aggregation for reference-based super-resolution. arXiv preprint arXiv:2201.04358

  13. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)

    Article  Google Scholar 

  14. Chen, H., Gu, J., Zhang, Z.: Attention in attention network for image super-resolution. arXiv preprint arXiv:2104.09497

  15. Xia, B., Hang, Y., Tian, Y., Yang, W., Liao, Q., Zhou, J.: Efficient non-local contrastive attention for image super-resolution. arXiv preprint arXiv:2201.03794

  16. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646–1654 (2016)

  17. Kim, J., Lee, J. K., Lee, K. M.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1637–1645 (2016)

  18. Lim,B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 136–144 (2017)

  19. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV) pp. 286–301 (2018)

  20. Hu, Y., Li, J., Huang, Y., Gao, X.: Channel-wise and spatial feature modulation network for single image super-resolution. IEEE Trans. Circuits Syst. Video Technol. 30(11), 3911–3927 (2019)

  21. Kim, J.-H., Choi, J.-H., Cheon, M., Lee, J.-S.: Ram: Residual attention module for single image super-resolution. arXiv preprint arXiv:1811.12043 (2018)

  22. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  23. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops pp. 136–144 (2017)

  24. Huang, J.-B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE conference on computer vision and pattern recognition pp. 5197–5206 (2015)

  25. Kingma Diederik, P., Adam, J. B.: A method for stochastic optimization. In: International conference on learning representations. ICLR

Download references

Acknowledgements

The paper is supported by The National Natural Science Foundation of China Nos. 42075139, 42077232. The Science and technology program of Jiangsu Province Nos. BE2020082 and BE2022063. The Innovation Fund of State Key Laboratory for Novel Software Technology No. ZZKT2022A18.

Author information

Authors and Affiliations

  1. State Key Lab. for Novel Software Technology, Nanjing University, Nanjing, 210023, China

    Shuang Wang, Zhengxing Sun & Qian Li

  2. Jiangsu Vocational Institute of Commerce, Nanjing, 211168, China

    Shuang Wang

  3. College of Meteorology and Oceanography, National University of Defense Technology, Changsha, China

    Qian Li

Authors
  1. Shuang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhengxing Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qian Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhengxing Sun.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, S., Sun, Z. & Li, Q. High-to-low-level feature matching and complementary information fusion for reference-based image super-resolution. Vis Comput 40, 99–108 (2024). https://doi.org/10.1007/s00371-023-02768-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02768-3

Keywords

Search

Navigation