Skip to main content
Springer Nature Link
Log in

A novel approach to multi-frame image super resolution using an innovative filter and pixel selection mechanism

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Deep learning-based methods excel at single image super resolution (SR), but struggle with multi-frame image SR due to their inability to effectively exploit the complementary information within low resolution (LR) images. Graph cuts remain effective in this context. A major challenge for graph cuts lies in the need for the energy function (EF) to adhere to specific regularization constraints. To meet these constraints, existing methods often rely on approximations that can degrade reconstruction quality. To minimize these negative effects, we design a filter to convert the EF into a standard form suitable for graph cuts and introduce a filter-based SR model with maximum accuracy. To fully utilize the complementary information within LR images, we also propose a LR pixel selection mechanism that selects and weights LR pixels in our model. Experimental results demonstrate the robustness of our model against noise and point spread function misestimation. Moreover, our model outperforms existing algorithms in reconstructing fine-grained details.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Availability of data and materials

Not applicable

References

  1. Cui, Y., Ren, W., Cao, X., Knoll, A.: Focal network for image restoration. In: IEEE/CVF International Conference on Computer Vision, pp. 13001–13011. IEEE (2023)

  2. Cui, Y., Ren, W., Yang, S., Cao, X., Knoll, A.: Irnext: rethinking convolutional network design for image restoration. In: International Conference on Machine Learning. (2023)

  3. Cui, Y., Tao, Y., Bing, Z., Ren, W., Gao, X., Cao, X., Huang, K., Knoll, A.: Selective frequency network for image restoration. In: International Conference on Learning Representations. (2023)

  4. Zhang, K., Liang, J., Gool, L.V., Timofte, R.: Designing a practical degradation model for deep blind image super-resolution. In: IEEE/CVF International Conference on Computer Vision, pp. 4791–4800. IEEE (2021)

  5. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695. IEEE (2022)

  6. Liang, J., Cao, J., Sun, G., Zhang, K., Gool, L.V., Timofte, R.: Swinir: image restoration using swin transformer. In: IEEE/CVF International Conference on Computer Vision, pp. 1833–1844. IEEE (2021)

  7. Jo, Y., Oh, S.W., Kang, J., Kim, S.J.: Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3224–3232. IEEE (2018)

  8. Zhang, A., Ren, W., Liu, Y., Cao, X.: Lightweight image super-resolution with superpixel token interaction. In: IEEE/CVF International Conference on Computer Vision, pp. 12728–12737. IEEE (2023)

  9. Rahiman, V.A., George, S.N.: Multi-frame image super resolution using spatially weighted total variation regularisations. IET Image Process. 14(10), 2187–2194 (2020)

    Article  MATH  Google Scholar 

  10. Zhu, X., Jin, P., Wang, X.X., Ai, N.: Multi-frame image super-resolution reconstruction via low-rank fusion combined with sparse coding. Multimed. Tools Appl. 78, 7143–7154 (2019)

    Article  MATH  Google Scholar 

  11. Mudenagudi, U., Banerjee, S., Kalra, P.K.: Space-time super-resolution using graph-cut optimization. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 995–1008 (2011)

    Article  MATH  Google Scholar 

  12. Zhang, D., Jodoin, P.M., Li, C., Wu, Y., Cai, G.: Novel graph cuts method for multi-frame super-resolution. IEEE Signal Process. Lett. 22(12), 2279–2283 (2015)

    Article  MATH  Google Scholar 

  13. Huang, S., Wu, J., Yang, Y., Lin, P.: Multi-frame image super-resolution reconstruction based on spatial information weighted fields of experts. Multidimens. Syst. Signal Process. 31, 1–20 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  14. Gharibi, Z., Faramarzi, S.: Multi-frame spatio-temporal super-resolution. SIViP 17, 4415–4424 (2023)

    Article  MATH  Google Scholar 

  15. Krainin, M., Liang, C.K., Levoy, M.S.: Handheld multi-frame super-resolution. ACM Trans. Gr. (TOG) 38(4), 1–18 (2019)

    Google Scholar 

  16. Honda, T., Sugimura, D., Hamamoto, T.: Multi-frame RGB/NIR imaging for low-light color image super-resolution. IEEE Trans. Comput. Imag. 6, 248–262 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  17. Baker, S., Kanade, T.: Limits on super-resolution and how to break them. IEEE Trans. Pattern Anal. Mach. Intell. 24(9), 1167–1183 (2002)

  18. Kolmogorov, V., Zabin, R.: What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004)

    Article  MATH  Google Scholar 

  19. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)

    Article  MATH  Google Scholar 

  20. Ghasemi-Falavarjani, N., Moallem, P., Rahimi, A.: Particle filter based multi-frame image super resolution. SIViP 17, 3247–3254 (2023)

  21. Raj, A., Zabih, R.: A graph cuts algorithm for generalized image deconvolution. In: IEEE International Conference on Computer Vision, pp. 1048–1054. IEEE (2005)

  22. Bergen, J.R., Anandan, P., Hanna, K.J., Hingorani, R.: Hierarchical model-based motion estimation. In: Lecture Notes in Computer Science (2005). https://doi.org/10.1007/3-540-55426-2_27

  23. Bätz, M., Eichenseer, A., Kaup, A.: Multi-image super-resolution using a dual weighting scheme based on Voronoi tessellation. In: IEEE International Conference on Image Processing, pp. 2822–2826. IEEE (2016)

  24. Bätz, M., Eichenseer, A., Kaup, A.: Multi-image super-resolution using a locally adaptive denoising-based refinement. In: IEEE 18th International Workshop on Multimedia Signal Processing, pp. 1–6. IEEE (2016)

  25. Wang, X., Chan, K.C.K., Yu, K., Dong, C., Loy, C.C.: Video restoration with enhanced deformable convolutional networks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 1954–1963. IEEE (2019)

  26. Nandi, D., Karmakar, J., Kumar, A., Mandal, M.K.: Sparse representation based multi-frame image super-resolution reconstruction using adaptive weighted features. IET Image Proc. 13(4), 663–672 (2019)

    Article  MATH  Google Scholar 

  27. Köhler, T., Huang, X., Schebesch, F., Aichert, A., Maier, A., Hornegger, J.: Robust multiframe super-resolution employing iteratively re-weighted minimization. IEEE Trans. Comput. Imag. 2(1), 42–58 (2016)

    Article  MathSciNet  Google Scholar 

  28. Milanfar, P.: MDSP Super-Resolution and Demosaicing Datasets. (2023). https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html. Accessed 18 Oct 2023

Download references

Acknowledgements

Not applicable.

Funding

This work was supported in part by the National Natural Science Foundation of China under Grant 42076159, in part by the Natural Science Foundation of Fujian Province under Grant 2022J011275 and 2021J06031, in part by the Doctoral Research Initiation Fund of Jimei University under Grant ZQ2023022, and in part by the Research Project of Fujian Association for Science and Technology Innovation Think Tank under Grant FJKX-2023XKB007.

Author information

Authors and Affiliations

  1. School of Science, Jimei University, Xiamen, 361021, Fujian, China

    Dongxiao Zhang, Ni Tang & Juhao Gao

Authors
  1. Dongxiao Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Ni Tang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Juhao Gao
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

All authors contributed to all parts of the manusripts.

Corresponding author

Correspondence to Dongxiao Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no Conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, D., Tang, N. & Gao, J. A novel approach to multi-frame image super resolution using an innovative filter and pixel selection mechanism. SIViP 19, 73 (2025). https://doi.org/10.1007/s11760-024-03687-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11760-024-03687-z

Keywords