Abstract
A skip-connection learning framework-based convolution neural network (CNN) has recently achieved great success in image super-resolution (SR). However, most CNN models based on the skip-connection learning framework do not fully make use of potential multi-scale features of images. In this paper, we propose a multi-scale skip-connection network (MSN) to improve the visual quality of the image SR. First, convolution kernels with different sizes are exploited to capture the multi-scale features of LR images. All the feature-maps captured by convolution kernels of the same size are direct input into a multi-scale hybrid group (MHG); second, the convolution layers of each MHG are composed of dilated convolutions and standard convolutions. The hybrid convolutions can fully train feature details obtained from preceding and current scale convolution layers; three, the output of each hybrid convolution layer is fed into subsequent hybrid convolution layers by skip-connections, thus producing dense connections; lastly, the meta-upscale module is used as the upscale module, which can magnify the trained feature maps arbitrary scale factors. By being evaluated on a wide variety of images, the proposed MSN network achieves an advantage over the state-of-the-art methods in terms of both numerical results and visual quality.
Similar content being viewed by others
References
Jia, B., Lv, J., Liu, D.: Deep learning-based automatic downbeat tracking: a brief review. Multimedia Syst. 25(6), 617–638 (2019)
Yan, C., Gong, B., Wei, Y., Gao, Y.: Deep multi-view enhancement hashing for image retrieval. IEEE Trans Pattern Analy Machine Intelligence 14(8), 1–8 (2020)
Li, Q., Li, Z., Jeon, G.W.G., Yang, X. M.: Gated multiple feedback network for image super-resolution. In BMVC (2019)
Zhang, L., Wu, X.: An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Process 15(8), 2226–2238 (2006)
Zhang, X., Wu, X.: Image interpolation by adaptive 2D autoregressive modeling and soft-decision estimation. IEEE Trans. Image Process. 17(6), 887–896 (2008)
Zhang, K., Gao, X., Tao, D., Li, X.: Single image super-resolution with non-local means and steering kernel regression. IEEE Trans. Image Process. 21(11), 4544–4556 (2012)
Li, X., He, H., Wang, R., Tao, D.: Single image super-resolution via directional group sparsity and directional features. IEEE Trans. Image Process 24(1), 327–340 (2015)
Song, M., Chen, C., Bu, J., Sha, T.: Image-based facial sketch-to-photo synthesis via online coupled dictionary learning. Inf. Sci. 193(6), 233–246 (2012)
Tao, D., Lin, X., Jin, L.: Principal component 2-D long short-term memory for font recognition on single Chinese characters. IEEE Trans. Cybern 46(3), 756–765 (2016)
Gao, X., Zhang, K., Tao, D., Li, X.: Image super-resolution with sparse neighbor embedding. IEEE Trans. Image Process. 21(7), 3194–3205 (2012)
Dong, C., Loy, C.C., He, K., Tang X.: Learning a deep convolutional network for image super-resolution. In ECCV (2014)
Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans Pattern Analy Mach Intell 38(2), 295–307 (2016)
Dong, C., Loy, C.C., Tang X.: Accelerating the super-resolution convolutional neural network. In ECCV (2016)
Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In CVPR (2016)
Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In CVPR (2016)
Han, S., Mao, H., Dally, W, J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR (2016) 2
Lim, B., Son, S., Kim, H., Nah, S., Lee, K. M.: Enhanced deep residual networks for single image super-resolution. In CVPRW (2017)
Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In CVPR (2017)
Huang, G., Liu, Z., Weinberger K. Q., van der Maaten and L.: Densely connected convolutional networks. In CVPR (2017)
Zhang, Y., Tian Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In CVPR (2018)
Hu, X.C., Mu, H.Y., Zhang, X.Y., Wang, Z.L., Sun, J., Tan, T.N.: Meta-SR: A magnification-arbitrary network for super-resolution. In CVPR (2019)
Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, D., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR (2016)
Tong, T., Li, G., Liu, X., & Gao, Q. Image super-resolution using dense skip connections. In ICCV (2017)
Zhang, Z.D., Wang, X.R., Jung, C.K.: DCSR: Dilated convolutions for single image super-resolution. IEEE Trans. Image Process. 28(4), 1625–1635 (2019)
Divakar, N., Babu, R. V.: Image denoising via CNNs: An Adversarial Approach. In CVPRW (2017)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z.Rethinking the inception architecture for computer vision. In CVPR (2016)
Li, X., Li, F., Fern, X., & Raich, R. Filter shaping for convolutional neural networks. In ICLR (2017)
Yu, F., & Koltun, V.: Multi-scale context aggregation by dilated convolutions. In ICLR (2016)
Ioffe, S., & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML (2015)
Glorot, X., Bordes, A., and Bengio, Y.: Deep sparse rectifier neural networks. In AISTATS (2011)
He, K., Zhang, Ren, S., Sun, J.: Deep residual learning for image recognition. In CVPR (2016)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Sheikh, H.R., Bovik, A.C., de Veciana, G.: An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans. Image Process. 14(12), 2117–2128 (2005)
Deng, J., Dong, W., Socher, R., Li, L. J., Li, F. F.: Imagenet: A large-scale hierarchical image database. In CVPR (2009)
Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H.: et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In CVPRW (2017)
Bevilacqua, M., Roumy, A., Guillemot, C., and Alberi- Morel M. L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC (2012)
Zeyde, R., Elad, M., and Protter, M.: On single image scale-up using sparse-representations. In Proc. 7th Int. Conf. Curves Surf, (2010)
Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV (2001)
Huang, J.B., Singh, A., and Ahuja N.: Single image super-resolution from transformed self-exemplars. In CVPR, (2015)
Matsui, Y., Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., Aizawa, K.: Sketch-based manga retrieval using manga109 dataset. Multimedia tools and applications 76(20), 21811–21838 (2017)
Kingma, D., and Ba Adam, J.: A method for stochastic optimization. In ICLR (2014)
Yan, C., Shao, B., Zhao, H., Ning, R., Zhang, Y., Xu, F.: 3D room layout estimation from a single RGB image. IEEE Trans. Multimedia (2020). https://doi.org/10.1109/TMM.2020.2967645
Yan, C., Li, Z., Zhang, Y., Liu, Y., Ji, X., and Zhang, Y.: Depth image denoising using nuclear norm and learning graph model. In arXiv:2008.03741, (2020), [online] Available: https://arxiv.org/abs/2008.03741
Li, Z., Li, Q., Wu, W., Yang, J.L., Yang, X.M.: Deep recursive up-down sampling networks for single image super-resolution. Neurocomputing 398(20), 377–388 (2020)
Acknowledgements
This work is supported by the National Natural Science Foundation of China under Grant 61472319, and by Natural Science Basic Research Plan in Shaanxi Province of China under Grant 2019JM-467.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Liu, J., Ge, J., Xue, Y. et al. Multi-scale skip-connection network for image super-resolution. Multimedia Systems 27, 821–836 (2021). https://doi.org/10.1007/s00530-020-00712-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00530-020-00712-2