Abstract
Single image super-resolution (SISR) is a fundamental image processing task, which aims to generate a high-resolution (HR) image from its low-resolution (LR) counterpart. Deep convolution neural networks (CNNs) have significantly improved the performance of SISR, and dominated the current research on SISR techniques. However, the performance improvement depends heavily on the size of the networks. In general, the deeper the networks, the better the performance, which limits their use to devices with limited computing and memory resources. The challenge is to find the optimal balance between network model complexity and SISR performance. In this paper, we propose a new lightweight SISR algorithm based on CNNs, which uses multi-path progressive feature fusion and attention mechanism. Our main contributions are as follows: (1) We propose a multi-path progressive feature fusion block (PFF), which can use the feature from the previous path to gradually guide the feature learning of the next path step by step in multiple paths. (2) We propose a multi-path feature attention mechanism (FAM), which can adaptively weigh the multi-path feature channels to be concatenated, improve the utilization of feature information and feature representation capability. The experimental results show that whether it is an objective measurement or a subjective measurement, our method is better than other similar state of the art methods, and has a better model complexity and performance balance.



Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request. The data we used are all publicly available, and the sources of all datasets can be found in the references of the original article.
References
Agarap A (2018) Deep learning using rectified linear units (relu). Preprint at arXiv:1803.08375v2
Ahn N, Kang B, Sohn KA (2018) Fast, accurate, and lightweight super-resolution with cascading residual network. In: 15th European Conference on computer vision (ECCV). Springer, Munich, pp 256–272. https://doi.org/10.1007/978-3-030-01249-6_16
Bevilacqua M, Roumy A, Guillemot C, et al (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: 23rd British Amchine Vision Conference (BMVC). https://doi.org/10.5244/C.26.135
Cao C, Liu X, Yang Y, et al (2015) Look and think twice: capturing top-down visual attention with feedback convolutional neural networks. In: IEEE International Conference on computer vision (ICCV). Springer, Santiago, pp 2956–2964. https://doi.org/10.1109/ICCV.2015.338
Dai T, Cai JR, Zhang YB, et al (2019) Second-order attention network for single image super-resolution. In: IEEE/CVF Conference on computer vision and pattern recognition (CVPR). IEEE, pp 11057–11066. https://doi.org/10.1109/CVPR.2019.01132
Dong C, Loy CC, He KM et al (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307. https://doi.org/10.1109/TPAMI.2015.2439281
Dong C, Loy CC, Tang XO (2016) Accelerating the super-resolution convolutional neural network. In: 14th European Conference on computer vision (ECCV). Springer, Amsterdam, pp 391–407. https://doi.org/10.1007/978-3-319-46475-6_257
Freeman WT, Pasztor EC, Carmichael OT (2000) Learning low-level vision. Int J Comput Vis 40(1):25–47. https://doi.org/10.1109/ICCV.1999.790414
Gao H, Yuan H, Wang Z et al (2019) Pixel transposed convolutional networks. IEEE Trans Pattern Anal Mach Intell 42(5):1218–1227. https://doi.org/10.1109/TPAMI.2019.2893965
Goodfellow I, Pouget-Abadie J, Mirza M, et al (2014) Generative adversarial nets. In: Ghahramani Z, Welling M, Cortes C, et al (eds) Advances in neural information processing systems, vol 27. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf. Accessed 17 Dec 2022
Han W, Chang S, Liu D, et al (2018) Image super-resolution via dual-state recurrent networks. In: 2018 IEEE Conference on computer vision and pattern recognition. IEEE, pp 1654–1663. https://doi.org/10.1109/CVPR.2018.00178
He K, Zhang X, Ren S, et al (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: 2015 IEEE International Conference on computer vision (ICCV). IEEE, pp 1026–1034. https://doi.org/10.1109/ICCV.2015.123
Howard AG, Zhu M, Chen B, et al (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. Preprint at arXiv:1704.04861
Hu J, Shen L, Sun G (2019) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 42:2011–2023. https://doi.org/10.1109/TPAMI.2019.2913372. (IEEE)
Huang G, Liu Z, Van Der Maaten L, et al (2017) Densely connected convolutional networks. In: 2017 IEEE Conference on computer vision and pattern recognition. IEEE, pp 2261–2269. https://doi.org/10.1109/CVPR.2017.243
Huang JB, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: 2015 IEEE Conference on computer vision and pattern recognition. IEEE, pp 5197–5206. https://doi.org/10.1109/CVPR.2015.7299156
Huang Z, Li W, Li J et al (2021) Dual-path attention network for single image super-resolution. Expert Syst Appl 169(114):450. https://doi.org/10.1016/j.eswa.2020.114450
Hui Z, Wang XM, Gao XB (2018) Fast and accurate single image super-resolution via information distillation network. In: 2018 IEEE/CVF Conference on computer vision and pattern recognition. IEEE, pp 723–731. https://doi.org/10.1109/CVPR.2018.00082
Hui Z, Gao X, Yang Y, et al (2019) Lightweight image super-resolution with information multi-distillation network. In: 27th ACM International Conference on multimedia. association for computing machinery, pp 2024–2032. https://doi.org/10.1145/3343031.3351084
Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on computer vision and pattern recognition. IEEE, pp 1646–1654. https://doi.org/10.1109/CVPR.2016.182
Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: 2016 IEEE Conference on computer vision and pattern recognition. IEEE, pp 1637–1645. https://doi.org/10.1109/CVPR.2016.181
Kim J, Lee JK, Lee KM (2017) Image super-resolution using dense skip connections. In: 2017 IEEE International Conference on computer vision. IEEE, pp 4799–4807. https://doi.org/10.1109/CVPR.2016.182
Kingma D, Ba J (2014) Adam: a method for stochastic optimization. Comput Sci Preprint at arXiv:1412.6980
Lai WS, Huang JB, Ahuja N, et al (2017) Deep Laplacian pyramid networks for fast and accurate super-resolution. In: 2017 IEEE Conference on computer vision and pattern recognition. IEEE, pp 5835–5843. https://doi.org/10.1109/CVPR.2017.618
Ledig C, Theis L, Huszar F, et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on computer vision and pattern recognition (CVPR)
Li W, Zhou K, Qi L, et al (2021) Lapar: inearly-assembled pixel-adaptive regression network for single image super-resolution and beyond. Adv Neural Inform Process Syst 33:20343–20355. Preprint at arXiv:2105.10422
Lim B, Son S, Kim H, et al (2017) Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on computer vision and pattern recognition workshops. IEEE, Honolulu, pp 136–144. https://doi.org/10.1109/CVPRW.2017.151
Liu J, Tang J, Wu G (2020) Residual feature distillation network for lightweight image super-resolution. In: Proceedings of the 16th European Conference on computer vision. Springer, Glasgow, pp 41–55. https://doi.org/10.1007/978-3-030-67070-2_2
Liu JJ, Hou Q, Cheng MM, et al (2020) Improving convolutional networks with self-calibrated convolutions. In: 2020 IEEE/CVF Conference on computer vision and pattern recognition. IEEE, pp 10096–10105. https://doi.org/10.1109/CVPR42600.2020.01011
Luo X, Xie Y, Zhang Y, et al (2020) Latticenet: towards lightweight image super-resolution with lattice block. In: 16th European Conference on computer vision. Springer, Glasgow, pp 272–289. https://doi.org/10.1007/978-3-030-58542-6_17
Martin D, Fowlkes C, Tal D, et al (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: 8th IEEE International Conference on computer vision, vol 2. IEEE, pp 416–423. https://doi.org/10.1109/ICCV.2001.937655
Matsui Y, Ito K, Aramaki Y et al (2017) Single image super-resolution from transformed self-exemplars. Multimed Tools Appl 76(20):21811–21838. https://doi.org/10.1007/s11042-016-4020-z
Muqeet A, Hwang J, Yang S, et al (2020) Multi-attention based ultra lightweight image super-resolution. In: 16th European Conference on computer vision. Springer, Glasgow, pp 103–118. https://doi.org/10.1007/978-3-030-67070-2_6
Park SC, Park MK, Kang MG (2003) Super-resolution image reconstruction a technical overview. IEEE Signal Process Mag 20(3):21–36. https://doi.org/10.1109/MSP.2003.1203207
Paszke A, Gross S, Chintala S et al (2017) Automatic differentiation in pytorch. 31st Conference on Neural Information Processing Systems (NIPS 2017), 4–9 December 2017, Long Beach, pp 1–4
Peyré G, Bougleux S, Cohen L (2008) Non-local regularization of inverse problems. In: European Conference on computer vision. Springer, Marseille, pp 57–68. https://doi.org/10.1007/978-3-540-88690-7_5
Shi WZ, Caballero J, Ledig C, et al (2013) Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In: International Conference on medical image computing and computer-assisted intervention. Springer, Nagoya, pp 9–16, https://doi.org/10.1007/978-3-642-40760-4_2
Sun L, Pan J, Tang J (2022) Shufflemixer: an efficient convnet for image super-resolution. In: Advances in Neural Information Processing Systems, vol 35
Tai Y, Liu XM, Xu CY (2017) Memnet: a persistent memory network for image restoration. In: 2017 IEEE International Conference on computer vision. IEEE, pp 4539–4547. https://doi.org/10.1109/ICCV.2017.486
Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: 2017 IEEE Conference on computer vision and pattern recognition. Springer, Munich, pp 3147–3155. https://doi.org/10.1109/CVPR.2017.298
Thornton WM, Atkinson PM, Holland DA (2006) Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping. Int J Remote Sens 27(3):473–491. https://doi.org/10.1080/01431160500207088
Timofte R, Agustsson E, Van Gool L, et al (2017) Ntire 2017 challenge on single image super-resolution: methods and results. In: 2017 IEEE Conference on computer vision and pattern recognition workshops. https://doi.org/10.1109/CVPRW.2017.149
Wang C, Li Z, Shi J (2019) Lightweight image super-resolution with adaptive weighted learning network. Preprint at arXiv:1904.02358
Wang X, Wang Q, Zhao Y, et al (2020) Lightweight single-image super-resolution network with attentive auxiliary feature learning. In: 15th Asian Conference on computer vision. Springer, Kyoto, pp 268–285. https://doi.org/10.1007/978-3-030-69532-3_17
Wang XT, Yu K, Wu SX, et al (2018) Esrgan: enhanced super-resolution generative adversarial networks. In: 2018 European Conference on computer vision (ECCV). Springer, Munich, pp 63–79. https://doi.org/10.1007/978-3-030-11021-5_5
Wang Z, Bovik AC, Sheikh HR et al (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612
Woo S, Park JY, Lee JY, et al (2018) Cbam: convolutional block attention module. In: 2018 European conference on computer vision (ECCV). Springer, Munich, pp 3–19. https://doi.org/10.1007/978-3-030-01234-2_1
Ymff0592 (2021) Super-resolution. https://github.com/ymff0592/super-resolution. Accessed 31 Mar 2022
Yu J, Fan Y, Huang T (2019) Wide activation for efficient image and video super-resolution. In: Sidorov K, Hicks Y (eds) Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, pp 52.1–52.13. https://doi.org/10.5244/C.33.52
Www Z, Yuen PC (2010) Very low resolution face recognition problem. Int J Remote Sens 21(1):327–340. https://doi.org/10.1109/BTAS.2010.5634490
Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: 7th International Conference on curves and surfaces. Springer, Avignon, pp 711–730. https://doi.org/10.1007/978-3-642-27413-8_47
Zhang J, Long C, Wang Y et al (2021) A two-stage attentive network for single image super-resolution. IEEE Trans Circ Syst Video Technol 32(3):1020–1033. https://doi.org/10.1109/TCSVT.2021.3071191
Zhang K, Zuo W, Zhang L (2018) Learning a single convolutional super-resolution network for multiple degradations. In: 2018 IEEE/CVF Conference on computer vision and pattern recognition. IEEE, pp 3262–3271. https://doi.org/10.1109/CVPR.2018.00344
Zhang X, Zhou X, Lin M, et al (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: 2018 IEEE/CVF conference on computer vision and pattern recognition. IEEE, pp 6848–6856. https://doi.org/10.1109/CVPR.2018.00716
Zhang YL, Li KP, Li K, et al (2018) Image super-resolution using very deep residual channel attention networks. In: 15th European Conference on computer vision (ECCV). Springer, Munich, pp 294–310. https://doi.org/10.1007/978-3-030-01234-2_18
Zhang YL, Li KP, Li K, et al (2018) Residual dense network for image super-resolution. In: 15th IEEE Conference on computer vision and pattern recognition. Springer, Munich, pp 294–310. https://doi.org/10.1007/978-3-030-01234-2_18
Zhao H, Kong X, He J, et al (2020) Efficient image super-resolution using pixel attentio. In: 16th European conference on computer vision. Springer, Glasgow, pp 56–72, https://doi.org/10.1007/978-3-030-67070-2_3
Zhao X, Liao Y, He T, et al (2019) F \(c^{2}\) n:fully channel-concatenated network for single image super-resolution. Preprint at arXiv:1907.03221v2
Zhou D, Duan R, Zhao L et al (2020) Single image super-resolution reconstruction based on multi-scale feature mapping adversarial network. Signal Process 166(107):251. https://doi.org/10.1016/j.sigpro.2019.107251
Zhou D, Chen Y, Li W et al (2021) Image super-resolution based on adaptive cascading attention network. Expert Syst Appl 186(115):815. https://doi.org/10.1016/j.eswa.2021.115815
Zhu X, Guo K, Ren S, et al (2022) Lightweight image super-resolution with expectation-maximization attention mechanism. IEEE Transactions on Circuits and Systems for Video Technology 32(3):1273C1284. https://doi.org/10.1109/TCSVT.2021.3078436
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Li, S., Zhou, D., Liu, Y. et al. Lightweight single image super-resolution based on multi-path progressive feature fusion and attention mechanism. Int. J. Mach. Learn. & Cyber. 14, 3517–3528 (2023). https://doi.org/10.1007/s13042-023-01847-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13042-023-01847-0