Skip to main content
Springer Nature Link
Log in

CEN-HDR: Computationally Efficient Neural Network for Real-Time High Dynamic Range Imaging

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 Workshops (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13802))

Included in the following conference series:

  • 2035 Accesses

Abstract

High dynamic range (HDR) imaging is still a challenging task in modern digital photography. Recent research proposes solutions that provide high-quality acquisition but at the cost of a very large number of operations and a slow inference time that prevent the implementation of these solutions on lightweight real-time systems. In this paper, we propose CEN-HDR, a new computationally efficient neural network by providing a novel architecture based on a light attention mechanism and sub-pixel convolution operations for real-time HDR imaging. We also provide an efficient training scheme by applying network compression using knowledge distillation. We performed extensive qualitative and quantitative comparisons to show that our approach produces competitive results in image quality while being faster than state-of-the-art solutions, allowing it to be practically deployed under real-time constraints. Experimental results show our method obtains a score of 43.04 \(\mu \)-PSNR on the Kalantari2017 dataset with a framerate of 33 FPS using a Macbook M1 NPU. The proposed network will be available at https://github.com/steven-tel/CEN-HDR

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Chen, Y., Kalantidis, Y., Li, J., Yan, S., Feng, J.: \(a^2\)-nets: double attention networks (2018). https://doi.org/10.48550/ARXIV.1810.11579. https://arxiv.org/abs/1810.11579

  2. Chollet, F.: Xception: deep learning with depthwise separable convolutions (2016). https://doi.org/10.48550/ARXIV.1610.02357. https://arxiv.org/abs/1610.02357

  3. Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp. 369–378. SIGGRAPH 1997, ACM Press/Addison-Wesley Publishing Co., USA (1997). https://doi.org/10.1145/258734.258884. https://doi.org/10.1145/258734.258884

  4. Froehlich, J., Grandinetti, S., Eberhardt, B., Walter, S., Schilling, A., Brendel, H.: Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays. In: Proceedings SPIE 9023 (2014). https://doi.org/10.1117/12.2040003

  5. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015). https://doi.org/10.48550/ARXIV.1503.02531. https://arxiv.org/abs/1503.02531

  6. Hu, J., Gallo, O., Pulli, K., Sun, X.: HDR deghosting: how to deal with saturation? In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1163–1170 (2013). https://doi.org/10.1109/CVPR.2013.154

  7. Kalantari, N.K., Ramamoorthi, R.: Deep high dynamic range imaging of dynamic scenes. ACM Trans. Graph. (Proceedings of SIGGRAPH 2017) 36(4), 1–12 (2017)

    Google Scholar 

  8. Khan, Z., Khanna, M., Raman, S.: FHDR: HDR image reconstruction from a single LDR image using feedback network (2019)

    Google Scholar 

  9. Li, F., et al.: Gamma-enhanced spatial attention network for efficient high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1032–1040 (2022)

    Google Scholar 

  10. Li, X., Wang, W., Hu, X., Yang, J.: Selective kernel networks (2019). https://doi.org/10.48550/ARXIV.1903.06586. https://arxiv.org/abs/1903.06586

  11. Liu, C.: Beyond pixels: exploring new representations and applications for motion analysis, Ph. D. thesis, MIT, USA (2009) aAI0822221

    Google Scholar 

  12. Liu, Y.L., et al.: Single-image HDR reconstruction by learning to reverse the camera pipeline (2020)

    Google Scholar 

  13. Liu, Z., et al.: Adnet: attention-guided deformable convolutional network for high dynamic range imaging (2021)

    Google Scholar 

  14. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  15. Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. In: ACM SIGGRAPH 2011 Papers. SIGGRAPH 2011, Association for Computing Machinery, New York, NY, USA (2011). https://doi.org/10.1145/1964921.1964935. https://doi.org/10.1145/1964921.1964935

  16. Mantiuk, R.K., Azimi, M.: Pu21: a novel perceptually uniform encoding for adapting existing quality metrics for HDR. In: 2021 Picture Coding Symposium (PCS), pp. 1–5 (2021). https://doi.org/10.1109/PCS50896.2021.9477471

  17. Nayar, S., Mitsunaga, T.: High dynamic range imaging: spatially varying pixel exposures. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2000 (Cat. No.PR00662), vol. 1, pp. 472–479 (2000). https://doi.org/10.1109/CVPR.2000.855857

  18. Niu, Y., Wu, J., Liu, W., Guo, W., Lau, R.W.H.: HDR-GAN: HDR image reconstruction from multi-exposed LDR images with large motions. IEEE Trans. Image Process. 30, 3885–3896 (2021). https://doi.org/10.1109/TIP.2021.3064433

    Article  Google Scholar 

  19. Park, J., Woo, S., Lee, J.Y., Kweon, I.S.: Bam: bottleneck attention module (2018). https://doi.org/10.48550/ARXIV.1807.06514. https://arxiv.org/abs/1807.06514

  20. Pérez-Pellitero, E., et al.: NTIRE 2022 challenge on high dynamic range imaging: methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1009–1023 (2022)

    Google Scholar 

  21. Prabhakar, K.R., Agrawal, S., Singh, D.K., Ashwath, B., Babu, R.V.: Towards practical and efficient high-resolution HDR Deghosting with CNN. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 497–513. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_30

    Chapter  Google Scholar 

  22. Prabhakar, K.R., Agrawal, S., Babu, R.V.: Self-gated memory recurrent network for efficient scalable HDR deghosting. CoRR abs/2112.13050 (2021). https://arxiv.org/abs/2112.13050

  23. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. CoRR abs/1505.04597 (2015). https://arxiv.org/abs/1505.04597

  24. Sen, P., Kalantari, N.K., Yaesoubi, M., Darabi, S., Goldman, D.B., Shechtman, E.: Robust patch-based HDR reconstruction of dynamic scenes. ACM Trans. Graphics (TOG) (Proceedings of SIGGRAPH Asia 2012) 31(6), 1–11 (2012)

    Google Scholar 

  25. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network (2016)

    Google Scholar 

  26. Tumblin, J., Agrawal, A., Raskar, R.: Why i want a gradient camera. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR2005), vol. 1, pp. 103–110 (2005). https://doi.org/10.1109/CVPR.2005.374

  27. Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: CBAM: convolutional block attention module (2018). https://doi.org/10.48550/ARXIV.1807.06521. https://arxiv.org/abs/1807.06521

  28. Wu, S., Xu, J., Tai, Y., Tang, C.: End-to-end deep HDR imaging with large foreground motions. CoRR abs/1711.08937 (2017). https://arxiv.org/abs/1711.08937

  29. Wu, S., Xu, J., Tai, Y.-W., Tang, C.-K.: Deep high dynamic range imaging with large foreground motions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 120–135. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_8

    Chapter  Google Scholar 

  30. Yan, Q., et al.: Attention-guided network for ghost-free high dynamic range imaging. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1751–1760 (2019)

    Google Scholar 

  31. Yan, Q., et al.: Multi-scale dense networks for deep high dynamic range imaging. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 41–50 (2019). https://doi.org/10.1109/WACV.2019.00012

  32. Yan, Q., et al.: Deep HDR imaging via a non-local network. IEEE Trans. Image Process. 29, 4308–4322 (2020). https://doi.org/10.1109/TIP.2020.2971346

    Article  MATH  Google Scholar 

  33. Yan, Q., et al.: A lightweight network for high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 824–832 (2022)

    Google Scholar 

  34. Ye, Q., Xiao, J., Lam, K., Okatani, T.: Progressive and selective fusion network for high dynamic range imaging. CoRR abs/2108.08585 (2021). https://arxiv.org/abs/2108.08585

  35. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions (2015). https://doi.org/10.48550/ARXIV.1511.07122. https://arxiv.org/abs/1511.07122

  36. Yu, G., Zhang, J., Ma, Z., Wang, H.: Efficient progressive high dynamic range image restoration via attention and alignment network (2022). https://doi.org/10.48550/ARXIV.2204.09213. https://arxiv.org/abs/2204.09213

  37. Zeng, H., Cai, J., Li, L., Cao, Z., Zhang, L.: Learning image-adaptive 3D lookup tables for high performance photo enhancement in real-time. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, p. 1 (2020). https://doi.org/10.1109/TPAMI.2020.3026740

  38. Zhang, H., Zu, K., Lu, J., Zou, Y., Meng, D.: EPSANet: an efficient pyramid squeeze attention block on convolutional neural network (2021). https://doi.org/10.48550/ARXIV.2105.14447. https://arxiv.org/abs/2105.14447

Download references

Author information

Authors and Affiliations

  1. ImViA EA7535, University Burgundy Franche-Comté, 21078, Dijon, France

    Steven Tel, Barthélémy Heyrman & Dominique Ginhac

Authors
  1. Steven Tel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Barthélémy Heyrman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dominique Ginhac
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Steven Tel .

Editor information

Editors and Affiliations

  1. IBM Research - MIT-IBM Watson AI Lab, Massachusetts, USA

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tel, S., Heyrman, B., Ginhac, D. (2023). CEN-HDR: Computationally Efficient Neural Network for Real-Time High Dynamic Range Imaging. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13802. Springer, Cham. https://doi.org/10.1007/978-3-031-25063-7_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25063-7_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25062-0

  • Online ISBN: 978-3-031-25063-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics