Abstract
Deep learning-based burst super-resolution (SR) approaches are extensively studied in recent years, prevailing in the synthetic datasets and the real datasets. However, the existing networks rarely pay attention to the enhanced denoising problem in raw domain and they are not sufficient to restore complex texture relationships between frames. In this paper, we propose a new framework named A RAW Burst Super-Resolution Method with Enhanced Denoising (EDRBSR), which solves the BurstSR problem by jointly denoising structure and reconstruction enhancement structure. We adopt a Denoising Network to further improve the performance of noise-free SR images. Also, we propose a Reconstruction Network to enhance spatial feature representation and eliminate the influence of spatial noise. In addition, we introduce a new pipeline to compensate for lost information. Experimental results demonstrate that our method over the existing state-of-the-art in both synthetic datasets and real datasets. Furthermore, our approach takes the 5th place in synthetic track of the NTIRE 2022 Burst Super-Resolution Challenge.
Q. Zheng and R. Gang—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abdelhamed, A., Lin, S., Brown, M.S.: A high-quality denoising dataset for smartphone cameras. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1692–1700 (2018)
Anaya, J., Barbu, A.: Renoir-a dataset for real low-light image noise reduction. J. Vis. Commun. Image Represent. 51, 144–154 (2018)
Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012)
Bhat, G., Danelljan, M., Timofte, R.: NTIRE 2021 challenge on burst super-resolution: methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 613–626 (2021)
Bhat, G., Danelljan, M., Van Gool, L., Timofte, R.: Deep burst super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9209–9218 (2021)
Burger, H.C., Schuler, C.J., Harmeling, S.: Image denoising: can plain neural networks compete with BM3D? In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2392–2399. IEEE (2012)
Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2004, vol. 1, pp. I-I. IEEE (2004)
Chen, C., Chen, Q., Do, M.N., Koltun, V.: Seeing motion in the dark. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3185–3194 (2019)
Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)
Dai, D., Timofte, R., Van Gool, L.: Jointly optimized regressors for image super-resolution. Comput. Graph. Forum 34, 95–104 (2015)
Dai, S., Han, M., Xu, W., Wu, Y., Gong, Y.: Soft edge smoothness prior for alpha channel super resolution. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2007)
Deudon, M., et al.: HighRes-net: recursive fusion for multi-frame super-resolution of satellite imagery (2020)
Farsiu, S., Elad, M., Milanfar, P.: Multiframe demosaicing and super-resolution of color images. IEEE Trans. Image Process. 15(1), 141–159 (2006)
Foi, A., Trimeche, M., Katkovnik, V., Egiazarian, K.: Practical poissonian-gaussian noise modeling and fitting for single-image raw-data. IEEE Trans. Image Process. 17(10), 1737–1754 (2008)
Fu, J., et al.: Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3146–3154 (2019)
Gharbi, M., Chaurasia, G., Paris, S., Durand, F.: Deep joint demosaicking and denoising. ACM Trans. Graph. (ToG) 35(6), 1–12 (2016)
Hardie, R.C.: High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system. Opt. Eng. 37(1), 247–260 (1998)
Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)
Ignatov, A., et al.: AIM 2019 challenge on raw to RGB mapping: methods and results. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3584–3590. IEEE (2019)
Ji, H., Fermüller, C.: Wavelet-based super-resolution reconstruction: theory and algorithm. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3954, pp. 295–307. Springer, Heidelberg (2006). https://doi.org/10.1007/11744085_23
Liu, J., Zhang, W., Tang, Y., Tang, J., Wu, G.: Residual feature aggregation network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2359–2368 (2020)
Luo, Z., et al.: EBSR: feature enhanced burst super-resolution with deformable alignment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 471–478 (2021)
Molini, A.B., Valsesia, D., Fracastoro, G., Magli, E.: DeepSUM: deep neural network for super-resolution of unregistered multitemporal images. IEEE Trans. Geosci. Remote Sens. 58(5), 3644–3656 (2019)
Muqeet, A., Hwang, J., Yang, S., Kang, J.H., Kim, Y., Bae, S.-H.: Multi-attention based ultra lightweight image super-resolution. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12537, pp. 103–118. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-67070-2_6
Shanmugam, D., Blalock, D., Balakrishnan, G., Guttag, J.: When and why test-time augmentation works. arXiv e-prints pp. arXiv-2011 (2020)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Tsai, R.: Multiframe image restoration and registration. Adv. Comput. Vis. Image Process. 1, 317–339 (1984)
Wang, X., Chan, K.C., Yu, K., Dong, C., Change Loy, C.: EDVR: video restoration with enhanced deformable convolutional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2019)
Wang, Y., Huang, H., Xu, Q., Liu, J., Liu, Y., Wang, J.: Practical deep raw image denoising on mobile devices. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12351, pp. 1–16. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58539-6_1
Wronski, B., et al.: Handheld multi-frame super-resolution. ACM Trans. Graph. (TOG) 38(4), 1–18 (2019)
Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Boissonnat, J.-D., et al. (eds.) Curves and Surfaces 2010. LNCS, vol. 6920, pp. 711–730. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-27413-8_47
Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856 (2018)
Acknowledgement
This paper is funded by the “Video Super-Resolution Algorithm Design and Software Development for Face Blur Problem” (JBKY20220210) and “Research and Simulation Experiment of Lightweight Sports Event Remote Production System” (ZZLX-2020-001) projects of the Academy of Broadcasting Science, National Radio and Television Administration.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zheng, Q. et al. (2022). A RAW Burst Super-Resolution Method with Enhanced Denoising. In: Yu, S., et al. Pattern Recognition and Computer Vision. PRCV 2022. Lecture Notes in Computer Science, vol 13537. Springer, Cham. https://doi.org/10.1007/978-3-031-18916-6_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-18916-6_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-18915-9
Online ISBN: 978-3-031-18916-6
eBook Packages: Computer ScienceComputer Science (R0)