Skip to main content
SpringerLink
Log in

A state-space super-resolution approach for video reconstruction

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The main objective of super-resolution video reconstruction is to make use of a set of low-resolution image frames to produce their respective counterparts with higher resolution. The conventional two-equation-based Kalman filter only considers the information from the previously reconstructed high-resolution frame and the currently observed low-resolution frame for producing each high-resolution frame. It has been observed that the information inherited in the previously observed low-resolution frame could be beneficial on the reconstruction of the super-resolution video. For that, an extra observation equation is incorporated into the framework of the conventional two-equation-based Kalman filtering in this paper to establish a three-equation-based state-space approach as a more generalized framework. The closed-form solution is mathematically derived, and extensive simulations using both artificially degraded and real-life image sequences are conducted to demonstrate its superior performance. Furthermore, a unified theoretical analysis is provided to analyze the relationship between the proposed framework and two existing super-resolution approaches, the sliding-window-based Bayesian estimation approach and the conventional two-equation-based Kalman filtering, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Chaudhuri S.: Super-resolution Imaging. Kluwer Academic, Dordrecht (2001)

    Google Scholar 

  2. Kang M.G., Chaudhuri S.: Super-resolution image reconstruction. IEEE Signal Process. Mag. 20, 19–20 (2003)

    Article  Google Scholar 

  3. Bose N.K., Chan R.H., Ng M.K.: Guest editorial. Int. J. Imaging Syst. Technol. 14, 35 (2004)

    Article  Google Scholar 

  4. Ng M.K., Chan T., Kang M.G., Milanfar P.: Super-resolution imaging: analysis, algorithms, and applications. EURASIP J. Appl. Signal Process. 2006, 1–2 (2006)

    Google Scholar 

  5. Baker S., Kanade T.: Limits on super-resolution and how to break them. IEEE Trans. Pattern Anal. Mach. Intell. 24, 1167–1183 (2002)

    Article  Google Scholar 

  6. Borman, S., Stevenson, R.L.: Spatial resolution enhancement of low-resolution image sequences: a comprehensive review with directions for future research. Technical Report, University of Notre Dame (1998)

  7. Tsai R.Y., Huang T.S.: Multiframe image restoration and registration. In: Huang, T.S.(eds) Advances in Computer Vision and Image Processing, JAI Press, Stanford (1984)

    Google Scholar 

  8. Bose, N.K., Kim, H.C., Valenzuela, H.M.: Recurcive implementation of total least squares algorithm for image reconstruction from noisy undersampled multiframes. In: Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 269–272, Minneapolis, MN (1993)

  9. Rajan D., Chaudhuri S.: Generalised interpolation and its application in super-resolution imaging. Image Vis. Comput. 19, 957–969 (2001)

    Article  Google Scholar 

  10. Patti A.J., Tekalp A.M.: Super resolution video reconstruction with arbitrary sampling lattices and nonzero aperture time. IEEE Trans. Image Process. 6, 1446–1451 (1997)

    Article  Google Scholar 

  11. Tom, B.C., Katsaggelos, A.K.: Reconstruction of a high-resolution image by simultaneous registration, restoration, and interpolation of low-resolution images. In: Proceedings of the IEEE International Conference on Image Processing, pp. 539–542 (1995)

  12. Schultz R.R., Stevenson R.L.: Extraction of high-resolution frames from video sequences. IEEE Trans. Image Process. 5, 996–1011 (1996)

    Article  Google Scholar 

  13. Hardie R.C., Barnard K.J., Armstrong E.E.: Joint MAP registration and high-resolution image estimation using a sequence of undersampled images. IEEE Trans. Image Process. 6, 1621–1633 (1997)

    Article  Google Scholar 

  14. Farsiu S., Robinson D., Elad M., Milanfar P.: Fast and robust multi-frame super-resolution. IEEE Trans. Image Process. 13, 1327–1344 (2004)

    Article  Google Scholar 

  15. He H., Kondi L.P.: An image super-resolution algorithm for different error levels per frame. IEEE Trans. Image Process. 15, 592–623 (2006)

    Article  Google Scholar 

  16. Bertero M., Boccacci P.: Introduction to Inverse Problems in Imaging. IOP, New York (1998)

    Book  MATH  Google Scholar 

  17. Freeman W.T., Pasztor E.C., Carmichael O.T.: Learning low-level vision. Int. J. Comput. Vis. 40, 25–47 (2000)

    Article  MATH  Google Scholar 

  18. Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: Proceedings of SIGGRAPH, pp. 327–340 (2001)

  19. Jiang, Z.D., Wong, T.T., Bao, H.J.: Practical super-resolution from dynamic video sequences. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pp. 549–554 (2003)

  20. Borman, S., Stevenson, R.L.: Simultaneous multi-frame MAP super-resolution video enhancement using spatio-temporal priors. In: Proceedings of IEEE International Conference on Image Processing. pp. 469–473 (1999)

  21. Zibetti M., Mayer J.: A robust and computationally efficient simultaneous super-resolution scheme for image sequences. IEEE Trans. Circuits Syst. Video Technol. 17, 1288–1300 (2007)

    Article  Google Scholar 

  22. Alvarez, L.D., Molina, R., Katsaggelos, A.K.: Multi-channel reconstruction of video sequences from low-resolution and compressed observations. In: The 8th Iberoamerican Congress on Pattern Recognition, pp. 46–53 (2003)

  23. Segall C.A., Molina R., Katsaggelos A.K.: High-resolution images from low-resolution compressed video. IEEE Signal Process. Mag. 20, 37–48 (2003)

    Article  Google Scholar 

  24. Kalman R.E.: A new approach to linear filtering and prediction problems. Trans. ASME J. Basic Eng. 82, 35–45 (1960)

    Google Scholar 

  25. Elad M., Feuer A.: Superresolution restoration of an image sequence: adaptive filtering approach. IEEE Trans. Image Process. 8, 387–395 (1999)

    Article  Google Scholar 

  26. Haykin S.: Adaptive Filter Theory. Prentice-Hall, Englewood Cliffs (1996)

    Google Scholar 

  27. Genest C., Zidek J.V.: Combining probability distributions: a critique and an annotated bibliography. Stat. Sci. 1, 114–148 (1986)

    Article  MathSciNet  Google Scholar 

  28. Ruanaidh J., Fitzgerald W.: Numerical Bayesian Methods Applied to Signal Processing, Statistics and Computing. Springer, Berlin (1996)

    Google Scholar 

  29. Golub G.H., Loan C.F.V.: Matrix Computations. Johns Hopkins University Press, Baltimore (1996)

    MATH  Google Scholar 

  30. Vandewalle, P., Susstrunk, S., Vetterli, M.: A frequency domain approach to registration of aliased images with application to super-resolutionn. EURASIP J. Appl. Signal Process. 2006, Article ID 71459 (2006)

  31. Schultz R.R., Li M., Stevenson R.L.: Subpixel motion estimation for super-resolution image sequence enhancement. J. Vis. Commun. Image Represent. 9, 38–50 (1998)

    Article  Google Scholar 

  32. Bertero M., Poggio T., Torre V.: Ill-posed problems in early vision. Proc. IEEE. 76, 869–889 (1988)

    Article  Google Scholar 

  33. Wang Z., Bovik A.C., Sheikh H.R., Simoncelli E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004)

    Article  Google Scholar 

  34. Reibman, A.R., Bell, R.M., Gray, S.: Quality assessment for super-resolution image enhancement. In: Proceedings of IEEE International Conference on Image Processing, pp. 2017–2020 (2006)

  35. Qu, Y., Pu, Z., Zhao, H., Zhao, Y.: Comparison of different quality assessment functions in autoregulative illumination intensity algorithms. Opt. Eng. 45, 117 201.1–117 201.6 (2006)

  36. Sheikh H.R., Bovik A.C., Cormack L.K.: No-reference quality assessment using natural scene statistics: JPEG 2000. IEEE Trans. Image Process. 14, 1918–1927 (2005)

    Article  Google Scholar 

  37. He, Y., Yap, K.-H., Chen, L., Chau, L.-P.: Blind super-resolution image reconstruction using a maximum a posteriori estimation. In: Proceedings of IEEE International Conference on Image Processing, pp. 1729–1732 (2006)

  38. Sroubek F., Cristobal G., Flusser J.: A unified approach to superresolution and multichannel blind deconvolution. IEEE Trans. Image Process. 16, 2322–2332 (2007)

    Article  MathSciNet  Google Scholar 

  39. Kay S.M.: Fundamentals of Statistical Signal Processing: Estimation Theory. Prentice-Hall, Englewood Cliffs (1993)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, 639798, Singapore

    Jing Tian & Kai-Kuang Ma

Authors
  1. Jing Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kai-Kuang Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kai-Kuang Ma.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Tian, J., Ma, KK. A state-space super-resolution approach for video reconstruction. SIViP 3, 217–240 (2009). https://doi.org/10.1007/s11760-008-0078-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-008-0078-z

Keywords

Search

Navigation