Skip to main content
SpringerLink
Log in

A linear well-posed solution to recover high-frequency information for super resolution image reconstruction

  • Published:
Multidimensional Systems and Signal Processing Aims and scope Submit manuscript

Abstract

Multiview super resolution image reconstruction (SRIR) is often cast as a resampling problem by merging non-redundant data from multiple images on a finer grid, while inverting the effect of the camera point spread function (PSF). One main problem with multiview methods is that resampling from nonuniform samples (provided by multiple images) and the inversion of the PSF are highly nonlinear and ill-posed problems. Non-linearity and ill-posedness are typically overcome by linearization and regularization, often through an iterative optimization process, which essentially trade off the very same information (i.e. high frequency) that we want to recover. We propose a different point of view for multiview SRIR that is very much like single-image methods which extrapolate the spectrum of one image selected as reference from among all views. However, for this, the proposed method relies on information provided by all other views, rather than prior constraints as in single-image methods which may not be an accurate source of information. This is made possible by deriving explicit closed-form expressions that define how the local high frequency information that we aim to recover for the reference high resolution image is related to the local low frequency information in the sequence of views. The locality of these expressions due to modeling using wavelets reduces the problem to an exact and linear set of equations that are well-posed and solved algebraically without requiring regularization or interpolation. Results and comparisons with recently published state-of-the-art methods show the superiority of the proposed solution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Antonini, M., Barlaud, M., Mathieu, P., & Daubechies, I. (1992). Image coding using wavelet transform. IEEE Transactions on Image Processing, 1(2), 205–220.

    Article  Google Scholar 

  • Babacan, S. D., Molina, R., & Katsaggelos, A. K. (2011). Variational Bayesian super resolution. IEEE Transactions on Image Processing, 20(4), 984–999.

    Article  MathSciNet  MATH  Google Scholar 

  • Bai, Z., Demmel, J., Dongarra, J., Ruhe, A., & van der Vorst, H. (2000). Templates for the solution of algebraic eigenvalue problems: A practical guide (Vol. 11). Philadelphia: SIAM.

    Book  MATH  Google Scholar 

  • Balci, M., & Foroosh, H. (2005a). Estimating sub-pixel shifts directly from the phase difference. In IEEE international conference on image processing, ICIP 2005 (Vol. 1) (pp I–1057). IEEE.

  • Balci, M., & Foroosh, H. (2005b). Inferring motion from the rank constraint of the phase matrix. In IEEE international conference on acoustics, speech, and signal processing, 2005. Proceedings (ICASSP’05) (Vol. 2) (pp. ii–925). IEEE.

  • Balci, M., & Foroosh, H. (2006a). Subpixel estimation of shifts directly in the Fourier domain. IEEE Transactions on Image Processing, 15(7), 1965–1972.

    Article  Google Scholar 

  • Balci, M., & Foroosh, H. (2006b). Subpixel registration directly from the phase difference. EURASIP Journal on Applied Signal Processing, 2006, 231–231.

  • Belekos, S. P., Galatsanos, N. P., & Katsaggelos, A. K. (2010). Maximum a posteriori video super-resolution using a new multichannel image prior. IEEE Transactions on Image Processing, 19(6), 1451–1464.

    Article  MathSciNet  MATH  Google Scholar 

  • Bishop, C. M., & Tipping, M. E. (2006). Bayesian image super resolution. US patent 7,106,914.

  • Candès, E. J., & Fernandez-Granda, C. (2014). Towards a mathematical theory of super-resolution. Communications on Pure and Applied Mathematics, 67(6), 906–956.

    Article  MathSciNet  MATH  Google Scholar 

  • Chan, R. H., Chan, T. F., Shen, L., & Shen, Z. (2003). Wavelet algorithms for high-resolution image reconstruction. SIAM Journal on Scientific Computing, 24(4), 1408–1432.

    Article  MathSciNet  MATH  Google Scholar 

  • Chu, K. E. (1987). The solution of the matrix equations axb- cxd= e and (ya- dz, yc- bz)=(e, f). Linear Algebra and Its Applications, 93, 93–105.

    Article  MathSciNet  MATH  Google Scholar 

  • Dai, Q., Yoo, S., Kappeler, A., & Katsaggelos, A. K. (2017). Sparse representation-based multiple frame video super-resolution. IEEE Transactions on Image Processing, 26(2), 765–781.

    Article  MathSciNet  Google Scholar 

  • Davis, G. M., & Nosratinia, A. (1999). Wavelet-based image coding: An overview. In Applied and computational control, signals, and circuits (pp. 369–434). Birkhäuser Boston.

  • Demirel, H., & Anbarjafari, G. (2011). Image resolution enhancement by using discrete and stationary wavelet decomposition. IEEE Transactions on Image Processing, 20(5), 1458–1460.

    Article  MathSciNet  MATH  Google Scholar 

  • de Lima, O., Vantaram, S. R., Piramanayagam, S., Saber, E., & Bengtson, K. R. (2013). An edge directed super resolution technique for multimedia applications. In: IS&T/SPIE electronic imaging, international society for optics and photonics (pp. 86630F–86630F).

  • Dong, W., Zhang, D., Shi, G., & Wu, X. (2011). Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Transactions on Image Processing, 20(7), 1838–1857.

    Article  MathSciNet  MATH  Google Scholar 

  • Elad, M., & Datsenko, D. (2009). Example-based regularization deployed to super-resolution reconstruction of a single image. The Computer Journal, 52(1), 15–30.

    Article  Google Scholar 

  • Elad, M., & Hel-Or, Y. (2001). A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur. IEEE Transactions on Image Processing, 10(8), 1187–1193.

    Article  MATH  Google Scholar 

  • Evangelidis, G. D., & Psarakis, E. Z. (2008). Parametric image alignment using enhanced correlation coefficient maximization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(10), 1858–1865.

    Article  Google Scholar 

  • Farsiu, S., Elad, M., & Milanfar, P. (2006). Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image Processing, 15(1), 141–159.

    Article  Google Scholar 

  • Farsiu, S., Robinson, M. D., Elad, M., & Milanfar, P. (2004). Fast and robust multiframe super resolution. IEEE Transactions on Image Processing, 13(10), 1327–1344.

    Article  Google Scholar 

  • Foroosh, H., & Balci, M. (2004) Sub-pixel registration and estimation of local shifts directly in the Fourier domain. In 2004 international conference on image processing. ICIP’04 (Vol. 3) (pp. 1915–1918). IEEE.

  • Foroosh, H., Zerubia, J. B., & Berthod, M. (2002). Extension of phase correlation to subpixel registration. IEEE Transactions on Image Processing, 11(3), 188–200.

    Article  Google Scholar 

  • Freeman, W. T., Jones, T. R., & Pasztor, E. C. (2002). Example-based super-resolution. Computer Graphics and Applications, IEEE, 22(2), 56–65.

    Article  Google Scholar 

  • Gajjar, P. P., & Joshi, M. V. (2010). New learning based super-resolution: Use of DWT and IGMRF prior. IEEE Transactions on Image Processing, 19(5), 1201–1213.

    Article  MathSciNet  MATH  Google Scholar 

  • Glasner, D., Bagon, S., & Irani, M. (2009) Super-resolution from a single image. In 2009 IEEE 12th international conference on computer vision (pp. 349–356). IEEE.

  • Hansen, P. C. (1987). The truncated svd as a method for regularization. BIT Numerical Mathematics, 27(4), 534–553.

    Article  MathSciNet  MATH  Google Scholar 

  • Huang, H., He, H., Fan, X., & Zhang, J. (2010). Super-resolution of human face image using canonical correlation analysis. Pattern Recognition, 43(7), 2532–2543.

    Article  MATH  Google Scholar 

  • Irani, M., & Peleg, S. (1991). Improving resolution by image registration. CVGIP: Graphical Models and Image Processing, 53(3), 231–239.

    Google Scholar 

  • Ji, H., & Fermuller, C. (2009). Robust wavelet-based super-resolution reconstruction: Theory and algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(4), 649–660.

    Article  Google Scholar 

  • Jiji, C., Joshi, M. V., & Chaudhuri, S. (2004). Single-frame image super-resolution using learned wavelet coefficients. International Journal of Imaging Systems and Technology, 14(3), 105–112.

    Article  Google Scholar 

  • Kappeler, A., Yoo, S., Dai, Q., & Katsaggelos, A. K. (2016). Video super-resolution with convolutional neural networks. IEEE Transactions on Computational Imaging, 2(2), 109–122.

    Article  MathSciNet  Google Scholar 

  • Kim, K., Min, J., Carlini, L., Unser, M., Manley, S., Jeon, D., & Ye, J. (2013). Fast maximum likelihood high-density low-SNR super-resolution localization microscopy. In 10th international conference on sampling theory and applications: Invited session VII: Sampling in bio imaging (Vol. 3). Bremen: Jacobs University.

  • Liu, Z., Nutter, B., & Mitra, S. (2012). Compressive sampling in fast wavelet-encoded MRI. In: 2012 IEEE southwest symposium on SSIAI (pp. 137–140). IEEE.

  • Lorette, A., Shekarforoush, H., & Zerubia, J. (1997) Super-resolution with adaptive regularization. In: 1997. Proceedings, international conference on image processing (Vol. 1) (pp. 169–172). IEEE.

  • Marquina, A., & Osher, S. J. (2008). Image super-resolution by TV-regularization and bregman iteration. Journal of Scientific Computing, 37(3), 367–382.

    Article  MathSciNet  MATH  Google Scholar 

  • Nguyen, N., & Milanfar, P. (2000). An efficient wavelet-based algorithm for image superresolution. In ICIP, IEEE (Vol. 2) (pp. 351–354).

  • Park, S. C., Park, M. K., & Kang, M. G. (2003). Super-resolution image reconstruction: A technical overview. IEEE Signal Processing Magazine, 20(3), 21–36.

    Article  Google Scholar 

  • Pickup, L. C., Capel, D. P., Roberts, S. J., & Zisserman, A. (2007). Overcoming registration uncertainty in image super-resolution: Maximize or marginalize? EURASIP Journal on Advances in Signal Processing, 2007(2), 20–20.

    MATH  Google Scholar 

  • Rivenson, Y., Stern, A., & Javidi, B. (2010). Single exposure super-resolution compressive imaging by double phase encoding. Optics Express, 18(14), 15094–15103.

    Article  Google Scholar 

  • Robinson, M. D., Toth, C. A., Lo, J. Y., & Farsiu, S. (2010). Efficient Fourier-wavelet super-resolution. IEEE Transactions on Image Processing, 19(10), 2669–2681.

    Article  MathSciNet  MATH  Google Scholar 

  • Serrai, H., & Senhadji, L. (2005). Acquisition time reduction in magnetic resonance spectroscopic imaging using discrete wavelet encoding. Journal of Magnetic Resonance, 177(1), 22–30.

    Article  Google Scholar 

  • Shekarforoush, H., Berthod, M., & Zerubia, J. (1996). Subpixel image registration by estimating the polyphase decomposition of cross power spectrum. In 1996. Proceedings CVPR’96, 1996 IEEE computer society conference on computer vision and pattern recognition (pp. 532–537). IEEE.

  • Shekarforoush, H., & Chellappa, R. (1999). Data-driven multichannel superresolution with application to video sequences. JOSA A, 16(3), 481–492.

    Article  Google Scholar 

  • Shen, H., Zhang, L., Huang, B., & Li, P. (2007). A map approach for joint motion estimation, segmentation, and super resolution. IEEE Transactions on Image Processing, 16(2), 479–490.

    Article  MathSciNet  Google Scholar 

  • Temizel, A., & Vlachos, T. (2005). Wavelet domain image resolution enhancement using cycle-spinning. Electronics Letters, 41(3), 119–121.

    Article  Google Scholar 

  • Tian, J., & Ma, K. K. (2011). A survey on super-resolution imaging. Signal, Image and Video Processing, 5(3), 329–342.

    Article  Google Scholar 

  • Tong, C., & Leung, K. (2007). Super-resolution reconstruction based on linear interpolation of wavelet coefficients. Multidimensional Systems and Signal Processing, 18(2–3), 153–171.

    Article  MathSciNet  MATH  Google Scholar 

  • Tsai, R., & Huang, T. S. (1984). Multiframe image restoration and registration. Advances in Computer Vision and Image Processing, 1(2), 317–339.

    Google Scholar 

  • Vandebril, R., Van Barel, M., & Mastronardi, N. (2007). Matrix computations and semiseparable matrices: Linear systems (Vol. 1). JHU Press.

  • Vandewalle, P., Sbaiz, L., Vandewalle, J., & Vetterli, M. (2007). Super-resolution from unregistered and totally aliased signals using subspace methods. IEEE Transactions on Signal Processing, 55(7), 3687–3703.

    Article  MathSciNet  MATH  Google Scholar 

  • Vandewalle, P., Süsstrunk, S., & Vetterli, M. (2006). A frequency domain approach to registration of aliased images with application to super-resolution. EURASIP Journal on Applied Signal Processing, 2006, 233–233.

    Google Scholar 

  • Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Article  Google Scholar 

  • Wu, W., Liu, Z., & He, X. (2011). Learning-based super resolution using kernel partial least squares. Image and Vision Computing, 29(6), 394–406.

    Article  Google Scholar 

  • Yang, J., Wright, J., Huang, T. S., & Ma, Y. (2010). Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 19(11), 2861–2873.

    Article  MathSciNet  MATH  Google Scholar 

  • Yuan, Q., Zhang, L., & Shen, H. (2012). Multiframe super-resolution employing a spatially weighted total variation model. IEEE Transactions on Circuits and Systems for Video Technology, 22(3), 379–392.

    Article  Google Scholar 

  • Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., & Zhang, L. (2016). Image super-resolution: The techniques, applications, and future. Signal Processing, 128, 389–408.

    Article  Google Scholar 

  • Zhang, K., Gao, X., Li, X., & Tao, D. (2011). Partially supervised neighbor embedding for example-based image super-resolution. IEEE Journal of Selected Topics in Signal Processing, 5(2), 230–239.

    Article  Google Scholar 

  • Zhao, S., Han, H., & Peng, S. (2003). Wavelet-domain HMT-based image super-resolution. In ICIP (Vol. 2) (pp. II–953). IEEE.

  • Zhou, F., Yang, W., & Liao, Q. (2012). Interpolation-based image super-resolution using multisurface fitting. IEEE Transactions on Image Processing, 21(7), 3312–3318.

    Article  MathSciNet  MATH  Google Scholar 

  • Zomet, A., Rav-Acha, A., & Peleg, S. (2001). Robust super-resolution. In CVPR (Vol. 1) (pp. I–645). IEEE.

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Central Florida, Orlando, FL, USA

    Vildan Atalay Aydin & Hassan Foroosh

Authors
  1. Vildan Atalay Aydin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hassan Foroosh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vildan Atalay Aydin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aydin, V.A., Foroosh, H. A linear well-posed solution to recover high-frequency information for super resolution image reconstruction. Multidim Syst Sign Process 29, 1309–1330 (2018). https://doi.org/10.1007/s11045-017-0499-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11045-017-0499-3

Keywords

Search

Navigation