Skip to main content
SpringerLink
Log in

Dictionary learning-based image super-resolution for multimedia devices

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In multimedia devices such as mobile phones, surveillance cameras, and web cameras, image sensors have limited spatial resolution. As a result, the image captured from these devices misses high-frequency content and exhibits visual artifacts. Image super-resolution (SR) algorithms can minimize these artifacts by reconstructing missing high-frequency textures. Image SR algorithm estimates a high resolution (HR) image from a given low-resolution (LR) image. Given a single LR image, reconstructing an HR image makes SR be an extremely ill-posed problem. Over the past decade, dictionary learning-based methods have shown promising results in SR reconstruction. These methods extract numerous patches from external images for training dictionaries via sparse representation. However, these methods do not involve any patch selection mechanism that enhances the learning process. This paper proposes a dictionary learning-based SR algorithm that extracts selective patches from an input LR image based on the iScore criterion. Results show that patch selection criteria keep only 36% of all extracted patches for training while improving the peak signal-to-noise ratio (PSNR). Furthermore, we have proposed a method to initialize dictionaries to achieve better convergence that enhances PSNR.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Algorithm 2
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

The datasets analysed during the current study are available in the GitHub repository, https://github.com/jbhuang0604/SelfExSR.

References

  1. Alam MS, Bognar JG, Hardie RC, Yasuda BJ (2000) Infrared image registration and high-resolution reconstruction using multiple translationally shifted aliased video frames. IEEE Trans Instrument Measurement 49(5):915–923. https://doi.org/10.1109/19.872908

    Article  Google Scholar 

  2. Amiri M, Ahmadyfard A, Abolghasemi V (2019) A fast video super resolution for facial image. Signal Process Image Commun 70:259–270

    Article  Google Scholar 

  3. Baker S, Kanade T (2002) Limits on super-resolution and how to break them. IEEE Trans Pattern Anal Mach Intell 24(9):1167–1183. https://doi.org/10.1109/TPAMI.2002.1033210

    Article  Google Scholar 

  4. Bevilacqua M, Roumy A, Guillemot C, Morel MLA (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. BMVA Press, https://doi.org/10.5244/C.26.135

  5. Bevilacqua M, Roumy A, Guillemot C, Morel M (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the 23rd british machine vision conference (BMVC). BMVA press, pp 135.1–135.10

  6. Cha Y, Kim S (2007) The error-amended sharp edge (ease) scheme for image zooming. IEEE Trans Image Process 16(6):1496–1505. https://doi.org/10.1109/TIP.2007.896645

    Article  MathSciNet  Google Scholar 

  7. Chang H, Yeung DY, Xiong Y (2004) Super-resolution through neighbor embedding. In: CVPR, vol 1, https://doi.org/10.1109/cvpr.2004.1315043

  8. Choi J-S, Kim M (2017) Single image super-resolution using global regression based on multiple local linear mappings. IEEE Trans Image Process 26 (3):1300–1314

    Article  MathSciNet  MATH  Google Scholar 

  9. Christensen-Jeffries K, Couture O, Dayton PA, Eldar YC, Hynynen K, Kiessling F, O’Reilly M, Pinton GF, Schmitz G, Tang M-X et al (2020) Super-resolution ultrasound imaging. Ultrasound Med Biology 46 (4):865–891

    Article  Google Scholar 

  10. Dai W, Xu T, Wang W (2012) Simultaneous codeword optimization (simco) for dictionary update and learning. IEEE Trans Signal Process 60 (12):6340–6353. https://doi.org/10.1109/TSP.2012.2215026

    Article  MathSciNet  MATH  Google Scholar 

  11. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38 (2):295–307. https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  12. Elad M (2010) Sparse and redundant representations: from theory to applications in signal and image processing. Springer, https://doi.org/10.1007/978-1-4419-7011-4

  13. Elad M, Figueiredo MA, Ma Y (2010) On the role of sparse and redundant representations in image processing. Proc IEEE 98(6):972–982. https://doi.org/10.1109/JPROC.2009.2037655

    Article  Google Scholar 

  14. Elad M, Hel-Or Y (2001) A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur. IEEE Transa Image Process 10(8):1187–1193. https://doi.org/10.1109/83.935034

    Article  MATH  Google Scholar 

  15. Farsiu S, Robinson MD, Elad M, Milanfar P (2004) Fast and robust multiframe super resolution. IEEE Trans Image Process 13(10):1327–1344. https://doi.org/10.1109/TIP.2004.834669

    Article  Google Scholar 

  16. Freeman WT, Jones TR, Pasztor EC (2002) Example-based super-resolution. IEEE Comput Graph Appl 22(2):56–65. https://doi.org/10.1109/38.988747

    Article  Google Scholar 

  17. Gao X, Zhang K, Tao D, Li X (2012) Image super-resolution with sparse neighbor embedding. IEEE Trans Image Process 21(7):3194–3205. https://doi.org/10.1109/TIP.2012.2190080

    Article  MathSciNet  MATH  Google Scholar 

  18. Glasner D, Bagon S, Irani M (2009) Super-resolution from a single image. In: 2009 IEEE 12Th international conference on computer vision, pp 349–356

  19. Hardie RC, Barnard KJ, Armstrong EE (1997) Joint map registration and high-resolution image estimation using a sequence of undersampled images. IEEE Trans Image Process 6(12):1621–1633. https://doi.org/10.1109/83.650116

    Article  Google Scholar 

  20. Irani M, Peleg S (1990) Super resolution from image sequences. In: [1990] Proceedings. 10th international conference on pattern recognition, vol ii, pp 115–120, https://doi.org/10.1109/icpr.1990.119340

  21. Jiang K, Wang Z, Yi P, Jiang J (2020) Hierarchical dense recursive network for image super-resolution. Pattern Recognit 107:107475. https://doi.org/10.1016/j.patcog.2020.107475

    Article  Google Scholar 

  22. Jiang K, Wang Z, Yi P, Wang G, Lu T, Jiang J (2019) Edge-enhanced GAN for remote sensing image superresolution. IEEE Trans Geosci Remote Sensing 57(8):5799–5812. https://doi.org/10.1109/TGRS.2019.2902431https://doi.org/10.1109/TGRS.2019.2902431

    Article  Google Scholar 

  23. Jiang C, Zhang Q, Fan R, Hu Z (2018) Super-resolution ct image reconstruction based on dictionary learning and sparse representation. Sci Rep 8(1):1–10

    Google Scholar 

  24. Joshi M, Jalobeanu A (2009) Map estimation for multiresolution fusion in remotely sensed images using an igmrf prior model. IEEE Trans Geosci Remote Sens 48(3):1245–1255

    Article  Google Scholar 

  25. Keren D, Peleg S, Brada R (1988) Image sequence enhancement using sub-pixel displacements. In: CVPR, vol 88, pp 5–9

  26. Keys RG (1981) Cubic convolution interpolation for digital image processing. IEEE Trans Acoustics, Speech, Signal Process 29(6):1153–1160. https://doi.org/10.1109/TASSP.1981.1163711

    Article  MathSciNet  MATH  Google Scholar 

  27. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654. https://doi.org/10.1109/CVPR.2016.182

  28. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4681–4690

  29. Li X, Cao G, Zhang Y, Shafique A, Fu P (2020) Combining synthesis sparse with analysis sparse for single image super-resolution. Signal Process Image Commun 115805:83

    Google Scholar 

  30. Li X, Hu Y, Gao X, Tao D, Ning B (2010) A multi-frame image super-resolution method. Signal Process 90(2):405–414

    Article  MATH  Google Scholar 

  31. Li M, Nguyen TQ (2008) Markov random field model-based edge-directed image interpolation. IEEE Trans Image Process 17(7):1121–1128. https://doi.org/10.1109/TIP.2008.924289

    Article  MathSciNet  Google Scholar 

  32. Li X, Orchard MT (2001) New edge-directed interpolation. IEEE Trans Image Process 10(10):1521–1527. https://doi.org/10.1109/83.951537

    Article  Google Scholar 

  33. Merino MT, Nunez J (2007) Super-resolution of remotely sensed images with variable-pixel linear reconstruction. IEEE Trans Geosci Remote Sens 45 (5):1446–1457

    Article  Google Scholar 

  34. Ning Q, Chen K, Yi L, Fan C, Lu Y, Wen J (2013) Image super-resolution via analysis sparse prior. IEEE Signal Process Lett 20 (4):399–402. https://doi.org/10.1109/LSP.2013.2242198

    Article  Google Scholar 

  35. Patel R (2020) Image super-resolution through quick learning from self-examples. In: Technologies for Sustainable Development: proceedings of the 7th Nirma University international conference on engineering (NUiCONE 2019). CRC Press, pp 161–166

  36. Patel R, Thakar V, Joshi R (2020) Single image super-resolution through sparse representation via coupled dictionary learning. Int J Electron Telecommun 66(2):347–353. https://doi.org/10.24425/ijet.2020.131884https://doi.org/10.24425/ijet.2020.131884

    Google Scholar 

  37. Peleg T, Elad M (2014) A statistical prediction model based on sparse representations for single image super-resolution. IEEE Trans Image Process 23(6):2569–2582. https://doi.org/10.1109/TIP.2014.2305844https://doi.org/10.1109/TIP.2014.2305844

    Article  MathSciNet  MATH  Google Scholar 

  38. Perez-Pellitero E, Salvador J, Ruiz-Hidalgo J, Rosenhahn B (2015) Accelerating super-resolution for 4k upscaling. In: 2015 IEEE international conference on consumer electronics (ICCE). IEEE, pp 317–320

  39. Starck J-L, Murtagh F, Fadili JM (2010) Sparse Image and Signal Processing: wavelets, curvelets, morphological diversity. Cambridge university press. https://doi.org/10.1017/CBO9780511730344

  40. Sun J, Zheng NN, Tao H, Shum HY (2003) Image hallucination with primal sketch priors. In: IEEE computer society conference on computer vision and pattern recognition. Proceedings, 2003, vol 2, pp II–729, https://doi.org/10.1109/cvpr.2003.1211539

  41. Thévenaz P, Blu T, Unser M (2000) Image interpolation and resampling. Handbook Med Imaging, Process Anal 1(1):393–420

    Article  Google Scholar 

  42. Timofte R, De Smet V, Van Gool L (2014) A+: Adjusted anchored neighborhood regression for fast super-resolution. In: Asian conference on computer vision. Springer, pp 111–126

  43. Tipping M, Bishop C (2002) Bayesian Image Super-Resolution. In: S. Becker, S. Thrun, K. Obermayer (eds) Advances in Neural Information Processing Systems. MIT Press https://proceedings.neurips.cc/paper/2002/???le/88bfcf02e7f554f9e9ea350b699bc6a7-Paper.pdf, vol 15

  44. Tropp JA, Gilbert AC (2007) Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans Inform Theory 53(12):4655–4666. https://doi.org/10.1109/TIT.2007.909108

    Article  MathSciNet  MATH  Google Scholar 

  45. Ur H, Gross D (1992) Improved resolution from subpixel shifted pictures. CVGIP: Graphical Models Image Process 54(2):181–186. https://doi.org/10.1016/1049-9652(92)90065-6

    Google Scholar 

  46. Van Reeth E, Tham IW, Tan CH, Poh CL (2012) Super-resolution in magnetic resonance imaging: a review. Concepts in Magnetic Resonance Part A 40(6):306–325

    Article  Google Scholar 

  47. Wang Q, Tang X, Shum H (2005) Patch based blind image super resolution. In: Tenth ieee international conference on computer vision (ICCV’05), vol I, pp 709–716. https://doi.org/10.1109/ICCV.2005.186

  48. Wang Z, Yi P, Jiang K, Jiang J, Han Z, Lu T, Ma J (2018) Multi-memory convolutional neural network for video super-resolution. IEEE Trans Image Process 28(5):2530–2544. https://doi.org/10.1109/TIP.2018.2887017https://doi.org/10.1109/TIP.2018.2887017

    Article  MathSciNet  Google Scholar 

  49. Wang S, Zhang L, Liang Y, Pan Q (2012) Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In: IEEE conference on computer vision and pattern recognition, pp 2216–2223. https://doi.org/10.1109/CVPR.2012.6247930

  50. Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873. https://doi.org/10.1109/TIP.2010.2050625

    Article  MathSciNet  MATH  Google Scholar 

  51. Yang W, Yuan T, Wang W, Zhou F, Liao Q (2017) Single-image super-resolution by subdictionary coding and kernel regression. IEEE Trans Syst, Man, Cybern: Syst 47(9):2478–2488. https://doi.org/10.1109/TSMC.2016.2523947https://doi.org/10.1109/TSMC.2016.2523947

    Google Scholar 

  52. Yedidia JS, Freeman WT, Weiss Y (2001) Generalized belief propagation. In: Advances in neural information processing systems, vol 13

  53. Yi P, Wang Z, Jiang K, Jiang J, Lu T, Ma J (2022) A progressive fusion generative adversarial network for realistic and consistent video super-resolution. IEEE Trans Pattern Anal Mach Intell 44(5):2264–2280. https://doi.org/10.1109/TPAMI.2020.3042298

    Google Scholar 

  54. Zeyde R, Elad M, Protter M (2012) On single image scale-up using sparse-representations. In: On single image scale-up using sparse representations. LNCS, vol 6920, pp 711–730. https://doi.org/10.1007/978-3-642-27413-8_47

  55. Zhang L, Wu X (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15 (8):2226–2238. https://doi.org/10.1109/TIP.2006.877407

    Article  Google Scholar 

  56. Zhang X, Wu X (2008) Image interpolation by adaptive 2-d autoregressive modeling and soft-decision estimation. IEEE Trans Image Process 17 (6):887–896. https://doi.org/10.1109/TIP.2008.924279

    Article  MathSciNet  Google Scholar 

Download references

Author information

Author notes

    Authors and Affiliations

    1. Electronics and Communication Engineering, Nirma University, Gota, Ahmedabad, 382481, Gujarat, India

      Rutul Patel

    2. Noble Group of Institutions, Gujarat Technological University, Junagadh Bhesan Road, Junagadh, 362310, Gujarat, India

      Vishvjit Thakar

    3. Electronics and Communication Engineering, Parul University, Waghodia, Vadodara, 391760, Gujarat, India

      Rutvij Joshi

    Authors
    1. Rutul Patel
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Vishvjit Thakar
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Rutvij Joshi
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Rutul Patel.

    Ethics declarations

    Conflict of Interests

    The authors have no competing interests to declare that are relevant to the content of this article.

    Additional information

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Vishvjit Thakar and Rutvij Joshi contributed equally to this work.

    Rights and permissions

    Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Patel, R., Thakar, V. & Joshi, R. Dictionary learning-based image super-resolution for multimedia devices. Multimed Tools Appl 82, 17243–17262 (2023). https://doi.org/10.1007/s11042-022-14076-4

    Download citation

    • Received:

    • Revised:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s11042-022-14076-4

    Keywords

    Search

    Navigation