Skip to main content
SpringerLink
Log in

Clustering based multiple branches deep networks for single image super-resolution

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Since the limitation of optical sensors, it’s often hard to obtain an image with the ideal resolution. Image super-resolution (SR) technology can generate a high-resolution image from the corresponding low-resolution image. Recently, deep learning (DL) based SR methods draw much attention due to their satisfying reconstruction results. However, these methods often neglect the diversity of image patches. Therefore, the reconstruction effect is limited. To fully exploit the texture variability across different image patches, we propose a universal, flexible, and effective framework. The proposed framework can be adopted to any DL based methods. It can significantly improve the SR accuracy while maintaining the running time. In the proposed framework, K-means is employed to cluster image patches into different categories. Multiple CNN branches are designed for these different categories to reconstruct the SR image. Each branch is weighted in accordance with the Euclidean distance to the cluster centers. Experimental results demonstrate that by applying the proposed framework, performance of the DL based SR method can be significantly improved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel ML (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding

  2. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  MATH  Google Scholar 

  3. Dong C, Loy CC, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: European conference on computer vision. Springer, pp 184–199

  4. Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. In: European conference on computer vision. Springer, pp 391–407

  5. Duchon CE (1979) Lanczos filtering in one and two dimensions. Japplmeteor 18(8):1016–1022

    Google Scholar 

  6. Farina R, Cuomo S, De Michele P, Piccialli F (2013) A smart gpu implementation of an elliptic kernel for an ocean global circulation model. Appl Math Sci 7(61-64):3007–3021

    Google Scholar 

  7. Fattal R (2007) Upsampling via imposed edges statistics. ACM Trans Graph (Proceedings of SIGGRAPH 2007), 26(3):to appear

  8. Freedman G, Fattal R (2011) Image and video upscaling from local self-examples. Acm Trans Graph 30(2):1–11

    Article  Google Scholar 

  9. Freeman WT, Jones TR, Pasztor EC (2002) Example-based super-resolution. IEEE Comput Graph Appl 22(2):56–65

    Article  Google Scholar 

  10. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  11. Haris M, Shakhnarovich G, Ukita N (2018) Deep backprojection networks for super-resolution. In: Conference on computer vision and pattern recognition

  12. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  13. Hou H, Andrews H (1978) Cubic splines for image interpolation and digital filtering. IEEE Trans Acoust Speech Signal Process 26(6):508–517

    Article  MATH  Google Scholar 

  14. Huang JB, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: Computer vision and pattern recognition, pp 5197–5206

  15. Irani M, Peleg S (1991) Improving resolution by image registration. CVGIP: Graph models image process 53(3):231–239

    Google Scholar 

  16. Jeon G, Anisetti M, Lee J, Bellandi V, Damiani E, Jeong J (2009) Concept of linguistic variable-based fuzzy ensemble approach: application to interlaced hdtv sequences. IEEE Trans Fuzzy Syst 17(6):1245–1258

    Article  Google Scholar 

  17. Jeon G, Anisetti M, Wang L, Damiani E (2016) Locally estimated heterogeneity property and its fuzzy filter application for deinterlacing. Inform Sci 354:112–130

    Article  Google Scholar 

  18. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision. Springer, pp 694–711

  19. Kim J, Kwon Lee J, Mu Lee K (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654

  20. Kim J, Kwon Lee J, Mu Lee K (2016) Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1637–1645

  21. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint

  22. Li M, Nguyen TQ (2008) Markov random field model-based edge-directed image interpolation. IEEE Trans Image Process 17(7):1121–1128

    Article  MathSciNet  Google Scholar 

  23. Li X, Orchard MT (2001) New edge-directed interpolation. IEEE Trans Image Process 10(10):1521–1527

    Article  Google Scholar 

  24. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: The IEEE conference on computer vision and pattern recognition (CVPR) workshops, vol 1, p 4

  25. Liu D, Wang Z, Wen B, Yang J, Han W, Huang TS (2016) Robust single image super-resolution via deep networks with sparse prior. IEEE Trans Image Process 25(7):3194–3207

    Article  MathSciNet  MATH  Google Scholar 

  26. Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Eighth IEEE International conference on computer vision, 2001. ICCV 2001. Proceedings, vol 2. IEEE, pp 416–423

  27. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in pytorch

  28. Piccialli F, Cuomo S, De Michele P (2013) A regularized mri image reconstruction based on hessian penalty term on cpu/gpu systems. Procedia Comput Sci 18:2643–2646

    Article  Google Scholar 

  29. Salvador J, Perez-Pellitero E (2015) Naive Bayes super-resolution forest. In: Proceedings of the IEEE International conference on computer vision, pp 325–333

  30. Schulter S, Leistner C, Bischof H (2015) Fast and accurate image upscaling with super-resolution forests. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3791– 3799

  31. Shi W, Caballero J, Huszár F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1874–1883

  32. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Computer Science

  33. Sun J, Sun J, Xu Z, Shum HY (2011) Gradient profile prior and its applications in image super-resolution and enhancement. IEEE Trans Image Process 20(6):1529–1542

    Article  MathSciNet  MATH  Google Scholar 

  34. Timofte R, De Smet V, Van Gool L (2013) Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of the IEEE international conference on computer vision, pp 1920–1927

  35. Timofte R, De Smet V, Van Gool L (2014) A+: Adjusted anchored neighborhood regression for fast super-resolution. In: Asian Conference on computer vision. Springer, pp 111–126

  36. Timofte R, Agustsson E, Van Gool L, Yang MH, Zhang L, Lim B, Son S, Kim H, Nah S, Lee KM et al (2017) Ntire 2017 challenge on single image super-resolution: methods and results. In: 2017 IEEE Conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 1110–1121

  37. Wang YQ (2014) A multilayer neural network for image demosaicking. In: 2014 IEEE International conference on image processing (ICIP). IEEE, pp 1852–1856

  38. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  39. Wang J, Wu J, Wu Z, Anisetti M, Jeon G (2018) Bayesian method application for color demosaicking. Opt Eng 57(5):053102

    Google Scholar 

  40. Wang X, Yu K, Dong C, Loy CC (2018) Recovering realistic texture in image super-resolution by deep spatial feature transform. arXiv:180402815

  41. Wu W, Zheng C (2013) Single image super-resolution using self-similarity and generalized nonlocal mean. In: TENCON 2013-2013 IEEE Region 10 conference (31194). IEEE, pp 1–4

  42. Wu W, Yang X, Liu K, Liu Y, Yan B, et al (2016) A new framework for remote sensing image super-resolution: sparse representation-based method by processing dictionaries with multi-type features. J Syst Archit 64:63–75

    Article  Google Scholar 

  43. Wu J, Anisetti M, Wu W, Damiani E, Jeon G (2016) Bayer demosaicking with polynomial interpolation. IEEE Trans Image Process 25(11):5369–5382

    Article  MathSciNet  MATH  Google Scholar 

  44. Yang CY, Huang JB, Yang MH (2010) Exploiting self-similarities for single frame super-resolution. In: Asian conference on computer vision, pp 497–510

  45. Yang J, Wang Z, Lin Z, Cohen S, Huang T (2012) Coupled dictionary training for image super-resolution. IEEE Trans Image Process 21(8):3467–3478

    Article  MathSciNet  MATH  Google Scholar 

  46. Yang X, Wu W, Liu K, Chen W, Zhang P, Zhou Z (2017) Multi-sensor image super-resolution with fuzzy cluster by using multi-scale and multi-view sparse coding for infrared image. Multimed Tools Appl 76(23):24871–24902

    Article  Google Scholar 

  47. Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: International conference on curves and surfaces. Springer, pp 711–730

  48. Zhang L, Wu X (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8):2226–2238

    Article  Google Scholar 

Download references

Acknowledgments

The research in our paper is sponsored by National Natural Science Foundation of China (No.61711540303, No.61701327), Science Foundation of Sichuan Science and Technology Department(No. 2018GZ0178).

Author information

Authors and Affiliations

  1. College of Electronics and Information Engineering, Sichuan University, Chengdu, Sichuan, 610064, China

    Zhen Li, Qilei Li, Wei Wu, Zongjun Wu, Lu Lu & Xiaomin Yang

Authors
  1. Zhen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qilei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zongjun Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lu Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaomin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Wu.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Z., Li, Q., Wu, W. et al. Clustering based multiple branches deep networks for single image super-resolution. Multimed Tools Appl 79, 9019–9035 (2020). https://doi.org/10.1007/s11042-018-7017-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-7017-y

Keywords

Search

Navigation