research-article

Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices

Authors:

Lei ZhangAuthors Info & Claims

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Pages 4034 - 4043

https://doi.org/10.1145/3474085.3475291

Published: 17 October 2021 Publication History

Abstract

Efficient and light-weight super resolution (SR) is highly demanded in practical applications. However, most of the existing studies focusing on reducing the number of model parameters and FLOPs may not necessarily lead to faster running speed on mobile devices. In this work, we propose a re-parameterizable building block, namely Edge-oriented Convolution Block (ECB), for efficient SR design. In the training stage, the ECB extracts features in multiple paths, including a normal 3 x 3 convolution, a channel expanding-and-squeezing convolution, and 1st-order and 2nd-order spatial derivatives from intermediate features. In the inference stage, the multiple operations can be merged into one single 3 3 convolution. ECB can be regarded as a drop-in replacement to improve the performance of normal 3 3 convolution without introducing any additional cost in the inference stage. We then propose an extremely efficient SR network for mobile devices based on ECB, namely ECBSR. Extensive experiments across five benchmark datasets demonstrate the effectiveness and efficiency of ECB and ECBSR. Our ECBSR achieves comparable PSNR/SSIM performance to state-of-the-art light-weight SR models, while it can super resolve images from 270p/540p to 1080p in real-time on commodity mobile devices, e.g., Snapdragon 865 SOC and Dimensity 1000+ SOC. The source code can be found at https://github.com/xindongzhang/ECBSR.

References

[1]

Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. 2018. Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European Conference on Computer Vision (ECCV). 252--268.

Digital Library

[2]

Sanjeev Arora, Nadav Cohen, and Elad Hazan. 2018. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning. PMLR, 244--253.

[3]

Roberto H Bamberger and Mark JT Smith. 1992. A filter bank for the directional decomposition of images: Theory and design. IEEE transactions on signal processing, Vol. 40, 4 (1992), 882--893.

Digital Library

[4]

Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. (2012).

[5]

Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017. Real-time video super-resolution with spatio-temporal networks and motion compensation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4778--4787.

[6]

Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759 (2014).

[7]

Xiangxiang Chu, Bo Zhang, Hailong Ma, Ruijun Xu, and Qingyuan Li. 2019. Fast, accurate and lightweight super-resolution with neural architecture search. arXiv preprint arXiv:1901.07261 (2019).

[8]

Xiangxiang Chu, Bo Zhang, and Ruijun Xu. 2020. Multi-objective reinforced evolution in mobile neural architecture search. In European Conference on Computer Vision. Springer, 99--113.

[9]

Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. 2019. Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1911--1920.

[10]

Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. 2021 a. Diverse Branch Block: Building a Convolution as an Inception-like Unit. arXiv preprint arXiv:2103.13425 (2021).

[11]

Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. 2021 b. RepVGG: Making VGG-style ConvNets Great Again. arXiv preprint arXiv:2101.03697 (2021).

[12]

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2015. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, Vol. 38, 2 (2015), 295--307.

Digital Library

[13]

Chao Dong, Chen Change Loy, and Xiaoou Tang. 2016. Accelerating the super-resolution convolutional neural network. In European conference on computer vision. Springer, 391--407.

[14]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.

[15]

Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5197--5206.

[16]

Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. 2019. Lightweight image super-resolution with information multi-distillation network. In Proceedings of the 27th ACM International Conference on Multimedia. 2024--2032.

Digital Library

[17]

Zheng Hui, Xiumei Wang, and Xinbo Gao. 2018. Fast and accurate single image super-resolution via information distillation network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 723--731.

[18]

Andrey Ignatov, Radu Timofte, Andrei Kulik, Seungsoo Yang, Ke Wang, Felix Baum, Max Wu, Lirong Xu, and Luc Van Gool. 2019. Ai benchmark: All about deep learning on smartphones in 2019. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). IEEE, 3617--3635.

[19]

Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. PMLR, 448--456.

Digital Library

[20]

Xiaotang Jiang, Huan Wang, Yiliu Chen, Ziqi Wu, Lichuan Wang, Bin Zou, Yafeng Yang, Zongyang Cui, Yu Cai, Tianhang Yu, et al. 2020. MNN: A universal and efficient inference engine. arXiv preprint arXiv:2002.12418 (2020).

[21]

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016a. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1646--1654.

[22]

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016b. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1637--1645.

[23]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[24]

Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2017. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition. 624--632.

[25]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature, Vol. 521, 7553 (2015), 436--444.

[26]

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4681--4690.

[27]

Royson Lee, Łukasz Dudziak, Mohamed Abdelfattah, Stylianos I Venieris, Hyeji Kim, Hongkai Wen, and Nicholas D Lane. 2020. Journey Towards Tiny Perceptual Super-Resolution. In European Conference on Computer Vision. Springer, 85--102.

[28]

Yawei Li, Shuhang Gu, Kai Zhang, Luc Van Gool, and Radu Timofte. 2020. Dhp: Differentiable meta pruning via hypernetworks. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part VIII 16. Springer, 608--624.

[29]

Yudong Liang, Jinjun Wang, Sanping Zhou, Yihong Gong, and Nanning Zheng. 2016. Incorporating image priors with deep convolutional neural networks for image super-resolution. Neurocomputing, Vol. 194 (2016), 340--347.

Digital Library

[30]

Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 136--144.

[31]

Jie Liu, Jie Tang, and Gangshan Wu. 2020. Residual feature distillation network for lightweight image super-resolution. arXiv preprint arXiv:2009.11551 (2020).

[32]

Cheng Ma, Yongming Rao, Yean Cheng, Ce Chen, Jiwen Lu, and Jie Zhou. 2020. Structure-preserving super resolution with gradient guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7769--7778.

[33]

Yinglan Ma, Hongyu Xiong, Zhe Hu, and Lizhuang Ma. 2019. Efficient super resolution using binarized neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 0--0.

[34]

David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vol. 2. IEEE, 416--423.

[35]

Ying Nie, Kai Han, Zhenhua Liu, An Xiao, Yiping Deng, Chunjing Xu, and Yunhe Wang. 2021. GhostSR: Learning Ghost Features for Efficient Image Super-Resolution. arXiv preprint arXiv:2101.08525 (2021).

[36]

Sylvain Paris, Samuel W Hasinoff, and Jan Kautz. 2011. Local Laplacian filters: Edge-aware image processing with a Laplacian pyramid. ACM Trans. Graph., Vol. 30, 4 (2011), 68.

Digital Library

[37]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, Vol. 32 (2019), 8026--8037.

Digital Library

[38]

Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. 2020. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10428--10436.

[39]

Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. 2018. Frame-recurrent video super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6626--6634.

[40]

Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1874--1883.

[41]

Dehua Song, Yunhe Wang, Hanting Chen, Chang Xu, Chunjing Xu, and DaCheng Tao. 2020. Addersr: Towards energy efficient image super-resolution. arXiv preprint arXiv:2009.08891 (2020).

[42]

Jian Sun, Zongben Xu, and Heung-Yeung Shum. 2008. Image super-resolution using gradient profile prior. In 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1--8.

[43]

Ying Tai, Jian Yang, and Xiaoming Liu. 2017. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3147--3155.

[44]

Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia. 2017. Detail-revealing deep video super-resolution. In Proceedings of the IEEE International Conference on Computer Vision. 4472--4480.

[45]

Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. 2017. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 114--125.

[46]

Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao. 2017. Image super-resolution using dense skip connections. In Proceedings of the IEEE international conference on computer vision. 4799--4807.

[47]

Chaofeng Wang, Zheng Li, and Jun Shi. 2019 b. Lightweight image super-resolution with adaptive weighted learning network. arXiv preprint arXiv:1904.02358 (2019).

[48]

Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. 2019 a. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 0--0.

[49]

Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2018. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 0--0.

[50]

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, Vol. 13, 4 (2004), 600--612.

Digital Library

[51]

Jingwei Xin, Nannan Wang, Xinrui Jiang, Jie Li, Heng Huang, and Xinbo Gao. 2020. Binarized neural network for single image super resolution. In European Conference on Computer Vision. Springer, 91--107.

[52]

Jiahui Yu, Yuchen Fan, and Thomas Huang. 2020. Wide activation for efficient image and video super-resolution. In 30th British Machine Vision Conference, BMVC 2019 .

[53]

Sergey Zagoruyko and Nikos Komodakis. 2017. Diracnets: Training very deep neural networks without skip-connections. arXiv preprint arXiv:1706.00388 (2017).

[54]

Roman Zeyde, Michael Elad, and Matan Protter. 2010. On single image scale-up using sparse-representations. In International conference on curves and surfaces. Springer, 711--730.

Digital Library

[55]

Kai Zhang, Martin Danelljan, Yawei Li, Radu Timofte, Jie Liu, Jie Tang, Gangshan Wu, Yu Zhu, Xiangyu He, Wenjie Xu, et al. 2020. AIM 2020 challenge on efficient super-resolution: Methods and results. In European Conference on Computer Vision. Springer, 5--40.

[56]

Kai Zhang, Shuhang Gu, Radu Timofte, Zheng Hui, Xiumei Wang, Xinbo Gao, Dongliang Xiong, Shuai Liu, Ruipeng Gang, Nan Nan, et al. 2019. Aim 2019 challenge on constrained super-resolution: Methods and results. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). IEEE, 3565--3574.

[57]

Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision (ECCV). 286--301.

Digital Library

[58]

Fuqiang Zhou, Xiaojie Li, and Zuoxin Li. 2018. High-frequency details enhancing DenseNet for super-resolution. Neurocomputing, Vol. 290 (2018), 34--42.

Digital Library

Cited By

Hong ZLiang GXiong L(2025)Gradient pooling distillation network for lightweight single image super-resolution reconstructionPeerJ Computer Science10.7717/peerj-cs.267911(e2679)Online publication date: 7-Feb-2025
https://doi.org/10.7717/peerj-cs.2679
Huang BWu LCao YZhong M(2025)Multi-scale information distillation attention network for super-resolution reconstruction of remote sensing imagesJournal of Measurements in Engineering10.21595/jme.2024.24351Online publication date: 8-Jan-2025
https://doi.org/10.21595/jme.2024.24351
Li AZhang LLiu YZhu C(2025)Exploring Frequency-Inspired Optimization in Transformer for Efficient Single Image Super-ResolutionIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2025.352992747:4(3141-3158)Online publication date: Apr-2025
https://doi.org/10.1109/TPAMI.2025.3529927
Show More Cited By

Index Terms

Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Reconstruction
  2. Computer graphics
    1. Image manipulation
      1. Computational photography

Recommendations

Fast Nearest Convolution for Real-Time Efficient Image Super-Resolution
Computer Vision – ECCV 2022 Workshops
Abstract
Deep learning-based single image super-resolution (SISR) approaches have drawn much attention and achieved remarkable success on modern advanced GPUs. However, most state-of-the-art methods require a huge number of parameters, memories, and ...
Image super-resolution by estimating the enhancement weight of self example and external missing patches

Image super-resolution (SR) is the process of generating a high-resolution (HR) image using one or more low-resolution (LR) inputs. Many SR methods have been proposed, but generating the small-scale structure of an SR image remains a challenging task. ...
Perception-oriented Single Image Super-Resolution Network with Receptive Field Block
Abstract
In recent years, deep learning has been widely applied to single image super-resolution(SISR). However, the majority of deep learning methods employ the Mean Square Error(MSE) loss as the objective optimization function, and the generated results ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

October 2021

5796 pages

ISBN:9781450386517

DOI:10.1145/3474085

General Chairs:
Heng Tao Shen
University of Electronic Science&Technology of China, China
,
Yueting Zhuang
Zhejiang University, China
,
John R. Smith
IBM, USA
,
Program Chairs:
Yang Yang
University of Electronic Science and Technology of China, China
,
Pablo Cesar
CWI&TU Delft, The Netherlands
,
Florian Metze
FACEBOOK, Inc., USA
,
Balakrishnan Prabhakaran
University of Texas at Dallas, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

The Hong Kong RGC RIF grant

Conference

MM '21

Sponsor:

SIGMM

MM '21: ACM Multimedia Conference

October 20 - 24, 2021

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

118
Total Citations
View Citations
926
Total Downloads

Downloads (Last 12 months)210
Downloads (Last 6 weeks)21

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hong ZLiang GXiong L(2025)Gradient pooling distillation network for lightweight single image super-resolution reconstructionPeerJ Computer Science10.7717/peerj-cs.267911(e2679)Online publication date: 7-Feb-2025
https://doi.org/10.7717/peerj-cs.2679
Huang BWu LCao YZhong M(2025)Multi-scale information distillation attention network for super-resolution reconstruction of remote sensing imagesJournal of Measurements in Engineering10.21595/jme.2024.24351Online publication date: 8-Jan-2025
https://doi.org/10.21595/jme.2024.24351
Li AZhang LLiu YZhu C(2025)Exploring Frequency-Inspired Optimization in Transformer for Efficient Single Image Super-ResolutionIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2025.352992747:4(3141-3158)Online publication date: Apr-2025
https://doi.org/10.1109/TPAMI.2025.3529927
Chai BWu DChen JYang MWang ZHu M(2025)REM: Enabling Real-Time Neural-Enhanced Video Streaming on Mobile Devices Using Macroblock-Aware Lookup TableIEEE Transactions on Mobile Computing10.1109/TMC.2024.349644324:3(2085-2097)Online publication date: Mar-2025
https://doi.org/10.1109/TMC.2024.3496443
Zhang KLi LJiao LLiu XMa WLiu FYang S(2025)CSCT: Channel–Spatial Coherent Transformer for Remote Sensing Image Super-ResolutionIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2025.354026063(1-14)Online publication date: 2025
https://doi.org/10.1109/TGRS.2025.3540260
Wang JXiang LLiu LXu JLi PXu QHe Z(2025)Toward Real-World Remote Sensing Image Super-Resolution: A New Benchmark and an Efficient ModelIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2024.351653863(1-13)Online publication date: 2025
https://doi.org/10.1109/TGRS.2024.3516538
Fan YLiu CTian RQian X(2025)InstanceSR: Efficient Reconstructing Small Object With Differential Instance-Level Super-ResolutionIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.349666435:3(2740-2752)Online publication date: Mar-2025
https://doi.org/10.1109/TCSVT.2024.3496664
Kairong CJun SBiao YMingzhi HJunlong Y(2025)A multi-scale enhanced large-kernel attention transformer network for lightweight image super-resolutionSignal, Image and Video Processing10.1007/s11760-024-03790-119:3Online publication date: 17-Jan-2025
https://doi.org/10.1007/s11760-024-03790-1
Zhang YTan WMao W(2025)Feature distillation network for efficient super-resolution with vast receptive fieldSignal, Image and Video Processing10.1007/s11760-024-03750-919:2Online publication date: 6-Jan-2025
https://doi.org/10.1007/s11760-024-03750-9
Dingyuan BBaoqing GTao RXingfang ZTao SYu WTao L(2025)F2RAIL: panoptic segmentation integrating Fpn and transFormer towards RAILwayApplied Intelligence10.1007/s10489-024-06158-755:4Online publication date: 9-Jan-2025
https://doi.org/10.1007/s10489-024-06158-7
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten