skip to main content
10.1145/3616901.3616912acmotherconferencesArticle/Chapter ViewAbstractPublication PagesfaimlConference Proceedingsconference-collections
research-article

Re-parameterizable Residual Multiple Convolutions Network for Efficient Single Image Super-Resolution

Published: 05 March 2024 Publication History

Abstract

The performance of single image super-resolution (SISR) has improved significantly in recent years. However, most existing methods suffer from large model parameters, high computational costs, and slow inference times, making them unsuitable for resource-constrained devices. To optimize these limitations, we propose a new re-parameterizable residual multiple convolutions network (RepRMCN) that strikes a better balance between model performance, scale, and inference time. Our approach introduces a convolutional reparameterization strategy that combines multiple convolutionals into a single convolution after model training, improving model performance during training and reducing the parameters and inference time when deployed. We also designed a simple grouped structure with different convolutions that inspired by grouped convolution, Inception, and heterogeneous kernel-based convolution. Our proposed structure splits features into two groups and processes them using re-parameterizable multiple convolutions module and dilation convolution, respectively. Experimental results show that RepRMCN outperforms existing high-efficiency SR models in terms of parameter reduction and inference time while maintaining good performance. Specifically, when tested on the Urban100 dataset with a 4× scale factor, our RepRMCN has 495K parameters, which is 48K less than the current state-of-the-art model RLFN. The inference time on the GeForce GTX 1080Ti is 16.47ms, which is comparable to RLFN.

References

[1]
Chao Dong, Chen Change Loy, Kaiming He and Xiaoou Tang. 2014. Learning a deep convolutional network for image super-resolution. (eds) Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science (), vol 8692. Springer, Cham. https://doi.org/10.1007/978-3-319-10593-2_13
[2]
Chao Dong, Chen Change Loy, and Xiaoou Tang. 2016. Accelerating the super-resolution convolutional neural network. (eds) Computer Vision – ECCV 2016. ECCV 2016. Lecture Notes in Computer Science (), vol 9906. Springer, Cham. https://doi.org/10.1007/978-3-319-46475-6_25, 2016:391-407
[3]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Las Vegas, NV, USA, 2016, pp. 1646-1654. https://doi.org/10.1109/CVPR.2016.181
[4]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Las Vegas, NV, USA, 2016, pp. 1637-1645. https://doi.org/10.1109/CVPR.2016.181
[5]
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2017. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Honolulu, HI, USA, 2017, pp. 5835-5843. https://10.1109/CVPR.2017.618
[6]
Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. 2018. Fast, accurate, and lightweight super-resolution with cascading residual network. (eds) Computer Vision – ECCV 2018. ECCV 2018. Lecture Notes in Computer Science (), vol 11214. Springer, Cham. https://doi.org/10.1007/978-3-030-01249-6_16
[7]
Zheng Hui, Xiumei Wang, and Xinbo Gao. 2018. Fast and accurate single image super-resolution via information distillation network. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Salt Lake City, UT, USA, 2018, pp. 723-731. https://doi.org/10.1109/CVPR.2018.00082
[8]
Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. 2019. Lightweight Image Super-Resolution with Information Multi-distillation Network. In Proceedings of the 27th ACM International Conference on Multimedia (MM '19). Association for Computing Machinery, New York, NY, USA, 2024–2032. https://doi.org/10.1145/3343031.3351084
[9]
Jie Liu, Wenjie Zhang, Yuting Tang, Jie Tang, and Gangshan Wu. 2020. Residual feature aggregation network for image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. IEEE, Seattle, WA, USA, 2020, pp. 2356-2365. https://10.1109/CVPR42600.2020.00243
[10]
Hengyuan Zhao, Xiangtao Kong, Jingwen He, Yu Qiao, and Chao Dong. 2020. Efficient image super-resolution using pixel attention. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science (), vol 12537. Springer, Cham. https://doi.org/10.1007/978-3-030-67070-2_3
[11]
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, International Conference on Computer Vision Workshops (ICCVW), Montreal, BC, Canada, 2021, pp. 1833-1844. https://doi.org/10.1109/ICCVW54120.2021.00210
[12]
Fangyuan Kong, Mingxi Li, Songwei Liu, Ding Liu, Jingwen He, Yang Bai, Fangmin Chen, and Lean Fu. 2022. Residual local feature network for efficient super-resolution. Proceedings of the IEEE/CVF international conference on computer vision. IEEE, New Orleans, LA, USA, 2022, pp. 765-775. https://doi.org/10.1109/CVPRW56347.2022.00092
[13]
Zheyuan Li, Yingqi Liu, Xiangyu Chen, Haoming Cai, Jinjin Gu, Yu Qiao, and Chao Dong. 2022. Blueprint separable residual network for efficient image super-resolution. In Proceedings of the IEEE/CVF international conference on computer vision. IEEE, New Orleans, LA, USA, 2022, pp. 832-842. https://doi.org/10.1109/CVPRW56347.2022.00099
[14]
Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. 2019. Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks. In Proceedings of the IEEE/CVF international conference on computer vision. IEEE, Seoul, Korea (South), 2019, pp. 1911-1920. https://doi.org/10.1109/ICCV.2019.00200
[15]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 6 (June 2017), 84–90. https://doi.org/10.1145/3065386
[16]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Boston, MA, USA, 2015, pp. 1-9. https://doi.org/10.1109/CVPR.2015.7298594
[17]
Pravendra Singh, Vinay Kumar Verma, Piyush Rai, and Vinay P. Namboodiri. 2019. Hetconv: Heterogeneous kernel-based convolutions for deep cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Long Beach, CA, USA, 2019, pp. 4830-4839. https://doi.org/10.1109/CVPR.2019.00497
[18]
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. 2021. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. IEEE, Nashville, TN, USA, 2021, pp. 13728-13737. https://doi.org/10.1109/CVPR46437.2021.01352
[19]
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, and Guiguang Ding. 2021. Diverse branch block: Building a convolution as an inception-like unit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Nashville, TN, USA, 2021, pp. 10881-10890. https://doi.org/10.1109/CVPR46437.2021.01074
[20]
Mu Hu, Junyi Feng, Jiashen Hua, Baisheng Lai, Jianqiang Huang, Xiaojin Gong, and Xiansheng Hua. 2022. Online convolutional re-parameterization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, New Orleans, LA, USA, 2022, pp. 558-567. https://doi.org/10.1109/CVPR52688.2022.00065.
[21]
Yawei Li, 2022. NTIRE 2022 challenge on efficient super-resolution: Methods and results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, New Orleans, LA, USA, 2022, pp. 1061-1101. https://doi.org/10.1109/CVPRW56347.2022.00118.
[22]
Laurent Sifre, and Stéphane Mallat. 2014. Rigid-motion scattering for texture classification. arXiv:1403.1687. Retrieved from https://arxiv.org/abs/1403.1687
[23]
Daniel Haase, and Manuel Amthor. 2020. Rethinking depthwise separable convolutions: How intra-kernel correlations lead to improved mobilenets. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. IEEE, Seattle, WA, USA, 2020, pp. 14588-14597. https://doi.org/10.1109/CVPR42600.2020.01461
[24]
Ying Tai, 2017. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE international conference on computer vision. IEEE, Venice, Italy, 2017, pp. 4549-4557. https://doi.org/10.1109/ICCV.2017.486
[25]
Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, and Xiaodi Hou; Garrison Cottrell. 2018. Understanding convolution for semantic segmentation. 2018 IEEE winter conference on applications of computer vision (WACV). IEEE, Lake Tahoe, NV, USA, 2018, pp. 1451-1460. https://doi.org/10.1109/WACV.2018.00163
[26]
Eirikur Agustsson, and Radu Timofte. 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. IEEE, Honolulu, HI, USA, 2017, pp. 1122-1131. https://doi.org/10.1109/CVPRW.2017.150
[27]
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. Proceedings of the IEEE conference on computer vision and pattern recognition workshops. IEEE, Honolulu, HI, USA, 2017, pp. 1132-1140. https://doi.org/10.1109/CVPRW.2017.151
[28]
Marco Bevilacqua, A. Roumy, Christine Guillemot, and Marie-Line Alberi-Morel. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings of the British Machine Vision Conference. 2012:135-1. https://doi.org/10.5244/C.26.135
[29]
Roman Zeyde, Michael Elad, and Matan Protter. 2012. On single image scale-up using sparse-representations. Curves and Surfaces 2010. Lecture Notes in Computer Science, vol 6920. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-27413-8_47
[30]
David Martin, Charless Fowlkes, D. Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001. IEEE, Vancouver, BC, Canada, 2001, pp. 416-423 vol.2.
[31]
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Boston, MA, USA, 2015, pp. 5197-5206. https://doi.org/10.1109/CVPR.2015.7299156
[32]
Abdul Muqeet, Jiwon Hwang, Subin Yang, JungHeum Kang, Yongwoo Kim, and Sung-Ho Bae. 2020. Multi-attention based ultra lightweight image super-resolution. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science (), vol 12537. Springer, Cham. https://doi.org/10.1007/978-3-030-67070-2_6

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
FAIML '23: Proceedings of the 2023 International Conference on Frontiers of Artificial Intelligence and Machine Learning
April 2023
296 pages
ISBN:9798400707544
DOI:10.1145/3616901
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 March 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Channel split
  2. Convolutional neural network
  3. Convolutional reparameterization
  4. Efficient super-resolution
  5. Grouped convolution

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • The Science and Technology planning Project of Guangdong Province

Conference

FAIML 2023

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 15
    Total Downloads
  • Downloads (Last 12 months)15
  • Downloads (Last 6 weeks)1
Reflects downloads up to 30 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media