Abstract
Mobile photography has made great strides in recent years. However, low light imaging remains a challenge. Long exposures can improve signal-to-noise ratio (SNR) but undesirable motion blur can occur when capturing dynamic scenes. Consequently, imaging pipelines often rely on computational photography to improve SNR by fusing multiple short exposures. Recent deep network-based methods have been shown to generate visually pleasing results by fusing these exposures in a sophisticated manner, but often at a higher computational cost.
We propose an end-to-end trainable burst denoising pipeline which jointly captures high-resolution and high-frequency deep features derived from wavelet transforms. In our model, precious local details are preserved in high-frequency sub-band features to enhance the final perceptual quality, while the low-frequency sub-band features carry structural information for faithful reconstruction and final objective quality. The model is designed to accommodate variable-length burst captures via temporal feature shifting while incurring only marginal computational overhead, and further trained with a realistic noise model for the generalization to real environments. Using these techniques, our method attains state-of-the-art performance on perceptual quality, while being an order of magnitude faster.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
A Deep Look into the iPhone’s new Deep Fusion Feature. https://tinyurl.com/deepfusion. Accessed 04 Nov 2019
Night Sight: Seeing in the Dark on Pixel Phones. https://tinyurl.com/googlenightsight. Accessed 04 Nov 2019
Buades, T., Lou, Y., Morel, J.M., Tang, Z.: A note on multi-image denoising. In 2009 International Workshop on Local and Non-Local Approximation in Image Processing, pp. 1–15. IEEE (2009)
Liu, C., Freeman, W.T.: A high-quality video denoising algorithm based on reliable motion estimation. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6313, pp. 706–719. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15558-1_51
Maggioni, M., Boracchi, G., Foi, A., Egiazarian, K.: Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms. IEEE Trans. Image Process. (TIP) 21(9), 3952–3966 (2012)
Liu, Z., Yuan, L., Tang, X., Uyttendaele, M., Sun, J.: Fast burst images denoising. ACM Trans. Graphics (TOG) 33(6), 232 (2014)
Godard, C., Matzen, K., Uyttendaele, M.: Deep burst denoising. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11219, pp. 560–577. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01267-0_33
Mildenhall, B., Barron, J.T., Chen, J., Sharlet, D., Ng, R., Carroll, R.: Burst denoising with kernel prediction networks. In: CVPR, pp. 2502–2510 (2018)
Aittala, M., Durand, F.: Burst image deblurring using permutation invariant convolutional neural networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 748–764. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01237-3_45
Hasinoff, S.W., et al.: Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graphics (TOG) 35(6), 192 (2016)
Kokkinos, F., Lefkimmiatis, S.: Iterative residual CNNS for burst photography applications. In: CVPR, pp. 5929–5938 (2019)
Dai, J., et al.: Deformable convolutional networks. In: ICCV, pp. 764–773 (2017)
Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs. In: CVPR, pp. 1712–1722 (2019)
Ke, T.W., Maire, M., Yu, S.X.: Multigrid neural architectures. In: CVPR, pp. 6665–6673 (2017)
Wang, J., et al.: Deep High-resolution Representation Learning for Visual Recognition. arXiv preprint arXiv:1908.07919 (2019)
Chen, Y., et al.: Drop an octave: reducing spatial redundancy in convolutional neural networks with octave convolution. arXiv preprint arXiv:1904.05049 (2019)
Blau, Y., Michaeli, T.: The perception-distortion tradeoff. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6228–6237 (2018)
Weickert, J.: Anisotropic diffusion in image processing. Teubner, Stuttgart (1998)
Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: ICCV, pp. 839–846 (1998)
Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60(1–4), 259–268 (1992)
Antonini, M., Barlaud, M., Mathieu, P., Daubechies, I.: Image coding using wavelet transform. IEEE Trans. Image Process. (TIP) 1(2), 205–220 (1992)
Portilla, J., Strela, V., Wainwright, M.J., Simoncelli, E.P.: Image denoising using scale mixtures of Gaussians in the wavelet domain. Trans. Img. Proc. 12(11), 1338–1351 (2003)
Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: CVPR, vol. 2, pp. 60–65. IEEE (2005)
Elad, M., Aharon, M.: Image denoising via learned dictionaries and sparse representation. In: CVPR, vol. 1, pp. 895–900. IEEE (2006)
Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. (TIP) 16, 2080–2095 (2007)
Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. (TIP) 26(7), 3142–3155 (2017)
Liu, P., Zhang, H., Zhang, K., Lin, L., Zuo, W.: Multi-level wavelet-CNN for image restoration. In: CVPR Workshop, pp. 773–782 (2018)
Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: CVPR, pp. 3929–3938 (2017)
Laine, S., Lehtinen, J., Aila, T.: High-quality self-supervised deep image denoising. arXiv preprint arXiv:1901.10277 (2019)
Batson, J., Royer, L.: Noise2Self: blind denoising by self-supervision. In: ICML, pp. 524–533 (2019)
Anwar, S., Barnes, N.: Real image denoising with feature attention. In: ICCV (2019)
Cha, S., Moon, T.: Fully convolutional pixel adaptive image denoiser. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4160–4169 (2019)
Gu, S., et al.: Self-guided network for fast image denoising. In: ICCV (2019)
Arias, P., Morel, J.M.: Video denoising via empirical Bayesian estimation of space-time patches. J. Math. Imaging Vis. 60(1), 70–93 (2018)
Xu, J., Huang, Y., Liu, L., Zhu, F., Hou, X., Shao, L.: Noisy-as-clean: learning unsupervised denoising from the corrupted image. arXiv preprint arXiv:1906.06878 (2019)
Wang, J.Z.: Wavelets and imaging informatics: a review of the literature. J. Biomed. Inform. 34(2), 129–141 (2001)
Williams, T., Li, R.: Wavelet pooling for convolutional neural networks. In: ICLR (2018)
Yoo, J., Uh, Y., Chun, S., Kang, B., Ha, J.W.: Photorealistic style transfer via wavelet transforms. In: ICCV (2019)
Deng, X., Yang, R., Xu, M., Dragotti, P.L.: Wavelet domain style transfer for an effective perception-distortion tradeoff in single image super-resolution (2019)
Kendall, A., Martirosyan, H., Dasgupta, S., Henry, P.: End-to-end learning of geometry and context for deep stereo regression. In: ICCV (2017)
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. In: NeurIPS, pp. 3391–3401 (2017)
Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. In: NeurIPS, pp. 5099–5108 (2017)
Lin, J., Gan, C., Han, S.: TSM: temporal shift module for efficient video understanding. In: ICCV (2019)
Jaroensri, R., Biscarrat, C., Aittala, M., Durand, F.: Generating training data for denoising real RGB images via camera pipeline simulation. arXiv preprint arXiv:1904.08825 (2019)
Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study. In: CVPR Workshop (2017)
Xue, T., Chen, B., Wu, J., Wei, D., Freeman, W.T.: Video enhancement with task-oriented flow. Int. J. Comput. Vis. (IJCV) 127(8), 1106–1125 (2019)
Xu, X., Li, M., Sun, W.: Learning deformable kernels for image and video denoising. arXiv preprint arXiv:1904.06903 (2019)
Steiner, B., et al.: PyTorch: An imperative style, high-performance deep learning library. NeurIPS 32 (2019)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)
Zhou, Y., et al.: When AWGN-based denoiser meets real noises. arXiv preprint arXiv:1904.03485 (2019)
Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: CVPR, pp. 3291–3300 (2018)
Chen, C., Chen, Q., Do, M., Koltun, V.: Seeing motion in the dark. In: ICCV (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Rong, X., Demandolx, D., Matzen, K., Chatterjee, P., Tian, Y. (2020). Burst Denoising via Temporally Shifted Wavelet Transforms. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12358. Springer, Cham. https://doi.org/10.1007/978-3-030-58601-0_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-58601-0_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58600-3
Online ISBN: 978-3-030-58601-0
eBook Packages: Computer ScienceComputer Science (R0)