Skip to main content
SpringerLink
Log in

Learned Reverse ISP with Soft Supervision

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 Workshops (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13802))

Included in the following conference series:

Abstract

RAW image serves as the foundation for camera imaging, which resides at the very beginning of the pipeline that generates sRGB images. Unfortunately, owing to special considerations, the information-rich RAW images are forfeited by default in most existing applications. To regain the RAW image, some works attempt to restore RAW images from RGB images. They focus on designing handcrafted model-based methods or complicated networks, however, ignoring the special property of RAW image, i.e., high dynamic range. To make up for this deficiency, we introduce a novel soft supervision, derived from the high dynamic range. Specifically, we propose to soften the original ground-truth as a multivariate Gaussian distribution so that networks could learn much more information. Then, we introduce a soft supervision driven network (SSDNet), based on convolution and transformer, for effectively restoring RAW images from RGB images. Quantitative and qualitative results show the promising restoration performance of RGB-to-RAW. In particular, our method achieved fifth place in the S7 track of AIM Reversed ISP Challenge. The source code will be available at https://github.com/yuezhang98/Learned-Reverse-ISP-with-Soft-Supervision.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Due to the limitation of computational resources, we have applied the convolution operation with maximum \(9\times 9\) kernel size.

References

  1. Abdelhamed, A., Lin, S., Brown, M.S.: A high-quality denoising dataset for smartphone cameras. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1692–1700 (2018)

    Google Scholar 

  2. Aharon, M., Elad, M., Bruckstein, A.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54(11), 4311–4322 (2006)

    Article  MATH  Google Scholar 

  3. Anwar, S., Barnes, N.: Real image denoising with feature attention. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3155–3164 (2019)

    Google Scholar 

  4. Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv:1607.06450 (2016)

  5. Brooks, T., Mildenhall, B., Xue, T., Chen, J., Sharlet, D., Barron, J.T.: Unprocessing images for learned raw denoising. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11036–11045 (2019)

    Google Scholar 

  6. Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR2005), vol. 2, pp. 60–65. IEEE (2005)

    Google Scholar 

  7. Burger, H.C., Schuler, C.J., Harmeling, S.: Image denoising: can plain neural networks compete with BM3D? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1256–1272 (2012)

    Google Scholar 

  8. Bychkovsky, V., Paris, S., Chan, E., Durand, F.: Learning photographic global tonal adjustment with a database of input/output image pairs. In: CVPR 2011, pp. 97–104. IEEE (2011)

    Google Scholar 

  9. Chen, Q., et al.: MixFormer: mixing features across windows and dimensions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5249–5259 (2022)

    Google Scholar 

  10. Chen, X., Wang, X., Zhou, J., Dong, C.: Activating more pixels in image super-resolution transformer. arXiv preprint arXiv:2205.04437 (2022)

  11. Chen, Y., Yu, W., Pock, T.: On learning optimized reaction diffusion processes for effective image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5261–5269 (2015)

    Google Scholar 

  12. Cheng, S., Wang, Y., Huang, H., Liu, D., Fan, H., Liu, S.: NBNet: noise basis learning for image denoising with subspace projection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4896–4906 (2021)

    Google Scholar 

  13. Conde, M.V., McDonagh, S., Maggioni, M., Leonardis, A., Pérez-Pellitero, E.: Model-based image signal processors via learnable dictionaries. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 481–489 (2022)

    Google Scholar 

  14. Conde, M.V., Timofte, R., et al.: Reversed image signal processing and raw reconstruction. AIM 2022 challenge report. In: Proceedings of the European Conference on Computer Vision Workshops (ECCVW) (2022)

    Google Scholar 

  15. Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)

    Article  MathSciNet  Google Scholar 

  16. Delbracio, M., Kelly, D., Brown, M.S., Milanfar, P.: Mobile computational photography: A tour. arXiv preprint arXiv:2102.09000 (2021)

  17. Gharbi, M., Chaurasia, G., Paris, S., Durand, F.: Deep joint demosaicking and denoising. ACM Trans. Graph. 35(6), 1–12 (2016)

    Google Scholar 

  18. Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2862–2869 (2014)

    Google Scholar 

  19. Guo, J., et al.: CMT: convolutional neural networks meet vision transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12175–12185 (2022)

    Google Scholar 

  20. Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1712–1722 (2019)

    Google Scholar 

  21. Han, K., et al.: A survey on vision transformer. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)

    Google Scholar 

  22. He, X., Cheng, J.: Revisiting L1 loss in super-resolution: a probabilistic view and beyond. arXiv preprint arXiv:2201.10084 (2022)

  23. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

    Google Scholar 

  24. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

  25. Kingma, D.P., Dhariwal, P.: Glow: Generative flow with invertible 1x1 convolutions. In: Advances in Neural Information Processing Systems 31 (2018)

    Google Scholar 

  26. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

    Google Scholar 

  27. Liu, P., Zhang, H., Zhang, K., Lin, L., Zuo, W.: Multi-level wavelet-CNN for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 773–782 (2018)

    Google Scholar 

  28. Liu, Y.L., et al.: Single-image HDR reconstruction by learning to reverse the camera pipeline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  29. Luo, W., Li, Y., Urtasun, R., Zemel, R.: Understanding the effective receptive field in deep convolutional neural networks. In: Advances in Neural Information Processing Systems 29 (2016)

    Google Scholar 

  30. Maini, R., Aggarwal, H.: A comprehensive review of image enhancement techniques. arXiv preprint arXiv:1003.4053 (2010)

  31. Nguyen, R.M., Brown, M.S.: Raw image reconstruction using a self-contained sRGB-jpeg image with only 64 KB overhead. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1655–1663 (2016)

    Google Scholar 

  32. Nguyen, R.M., Brown, M.S.: Raw image reconstruction using a self-contained sRGB-JPEG image with small memory overhead. Int. J. Comput. Vision 126(6), 637–650 (2018)

    Article  MathSciNet  Google Scholar 

  33. Plötz, T., Roth, S.: Neural nearest neighbors networks. In: Advances in Neural information processing systems 31 (2018)

    Google Scholar 

  34. Punnappurath, A., Brown, M.S.: Learning raw image reconstruction-aware deep image compressors. IEEE Trans. Pattern Anal. Mach. Intell. 42(4), 1013–1019 (2019)

    Article  Google Scholar 

  35. Purohit, K., Rajagopalan, A.: Region-adaptive dense network for efficient motion deblurring. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 11882–11889 (2020)

    Google Scholar 

  36. Qian, G., Gu, J., Ren, J.S., Dong, C., Zhao, F., Lin, J.: Trinity of pixel enhancement: a joint solution for demosaicking, denoising and super-resolution. arXiv preprint arXiv:1905.02538 (2019)

  37. Reddi, S.J., Kale, S., Kumar, S.: On the convergence of Adam and beyond. arXiv preprint arXiv:1904.09237 (2019)

  38. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  39. Roth, S., Black, M.J.: Fields of experts. Int. J. Comput. Vision 82(2), 205–229 (2009)

    Article  MATH  Google Scholar 

  40. Schwartz, E., Giryes, R., Bronstein, A.M.: DeepISP: toward learning an end-to-end image processing pipeline. IEEE Trans. Image Process. 28(2), 912–923 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  41. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

    Google Scholar 

  42. Talebi, H., Milanfar, P.: Global image denoising. IEEE Trans. Image Process. 23(2), 755–768 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  43. Tian, C., Fei, L., Zheng, W., Xu, Y., Zuo, W., Lin, C.W.: Deep learning on image denoising: an overview. In: Neural Networks (2020)

    Google Scholar 

  44. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30 (2017)

    Google Scholar 

  45. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  46. Wronski, B., et al.: Handheld multi-frame super-resolution. ACM Trans. Graph. (TOG) 38(4), 1–18 (2019)

    Article  Google Scholar 

  47. Xia, M., Liu, X., Wong, T.T.: Invertible grayscale. ACM Trans. Graph. (TOG) 37(6), 1–10 (2018)

    Article  Google Scholar 

  48. Xing, Y., Qian, Z., Chen, Q.: Invertible image signal processing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6287–6296 (2021)

    Google Scholar 

  49. Yue, Z., Yong, H., Zhao, Q., Zhang, L., Meng, D.: Variational denoising network: toward blind noise modeling and removal. arXiv preprint arXiv:1908.11314 (2019)

  50. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5728–5739 (2022)

    Google Scholar 

  51. Zamir, S.W., et al.: CycleISP: real image restoration via improved data synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2696–2705 (2020)

    Google Scholar 

  52. Zamir, S.W., et al.: Learning enriched features for real image restoration and enhancement. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12370, pp. 492–511. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58595-2_30

    Chapter  Google Scholar 

  53. Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  54. Zhang, K., Zuo, W., Zhang, L.: FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. 27(9), 4608–4622 (2018)

    Article  MathSciNet  Google Scholar 

  55. Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: 2011 International Conference on Computer Vision, pp. 479–486. IEEE (2011)

    Google Scholar 

Download references

Acknowledgment

This work was supported by the National Key R &D Program of China under Grant 2018AAA0102100.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Central South University, Changsha, 410083, China

    Beiji Zou & Yue Zhang

  2. Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Changsha, 410083, China

    Beiji Zou & Yue Zhang

Authors
  1. Beiji Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yue Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Beiji Zou .

Editor information

Editors and Affiliations

  1. IBM Research - MIT-IBM Watson AI Lab, Massachusetts, USA

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 123 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zou, B., Zhang, Y. (2023). Learned Reverse ISP with Soft Supervision. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13802. Springer, Cham. https://doi.org/10.1007/978-3-031-25063-7_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25063-7_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25062-0

  • Online ISBN: 978-3-031-25063-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation