Skip to main content
SpringerLink
Log in

A Multi-purpose Convolutional Neural Network for Simultaneous Super-Resolution and High Dynamic Range Image Reconstruction

  • Conference paper
  • First Online:
Computer Vision – ACCV 2018 (ACCV 2018)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11363))

Included in the following conference series:

Abstract

High dynamic range (HDR) UHD-TVs are being rapidly deployed in consumer markets, offering a highly realistic experience to customers. However, these HDR UHD-TVs still need to handle the legacy low resolution (LR) video of standard dynamic range (SDR). In this paper, we propose a convolutional neural network based structure for the joint learning of super-resolution and inverse tone-mapping, which can be used for converting LR-SDR legacy video to high resolution (HR) HDR video. Our proposed structure is designed to perform three tasks: (i) SDR-to-HDR conversion of LR images, (ii) super-resolution of LR-SDR images to HR-SDR images and (iii) joint conversion from LR-SDR to HR-HDR images. We show the effectiveness of our proposed joint learning CNN architecture with extensive experiments.

This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00419, Intelligent High Realistic Visual Processing for Smart Broadcasting Media).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Banterle, F., Ledda, P., Debattista, K., Chalmers, A., Bloj, M.: A framework for inverse tone mapping. Vis. Comput. 23(7), 467–478 (2007)

    Article  Google Scholar 

  2. Bengtsson, T., Gu, I.Y.H., Viberg, M., Lindström, K.: Regularized optimization for joint super-resolution and high dynamic range image reconstruction in a perceptually uniform domain. In: 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1097–1100. IEEE (2012)

    Google Scholar 

  3. Bengtsson, T., McKelvey, T., Gu, I.Y.H.: Super-resolution reconstruction of high dynamic range images with perceptual weighting of errors. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2212–2216. IEEE (2013)

    Google Scholar 

  4. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  5. Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R.K., Unger, J.: HDR image reconstruction from a single exposure using deep CNNs. ACM Trans. Graph. 36(6), 178 (2017)

    Article  Google Scholar 

  6. Endo, Y., Kanamori, Y., Mitani, J.: Deep reverse tone mapping. ACM Trans. Graph. 36(6), 177 (2017)

    Article  Google Scholar 

  7. Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 349–356. IEEE (2009)

    Google Scholar 

  8. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)

    Google Scholar 

  9. Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 315–323 (2011)

    Google Scholar 

  10. Gunturk, B.K., Gevrekci, M.: High-resolution image reconstruction from multiple differently exposed images. IEEE Signal Process. Lett. 13(4), 197–200 (2006)

    Article  Google Scholar 

  11. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)

    Google Scholar 

  12. Huo, Y., Yang, F., Dong, L., Brost, V.: Physiological inverse tone mapping based on retina response. Vis. Comput. 30(5), 507–517 (2014)

    Article  Google Scholar 

  13. ITU-R: Parameter values for the HDTV standards for production and international programme exchange. ITU-R Rec. BT.709-5 (2002). http://www.itu.int/rec/R-REC-BT.709

  14. ITU-R: Reference electro-optical transfer function for flat panel displays used in HDTV studio production. ITU-R Rec. BT.1886 (2011)

    Google Scholar 

  15. ITU-R: Parameter values for ultra-high definition television systems for production and international programme exchange. Document ITU-R Rec. BT.2020-1 (2014). http://www.itu.int/rec/R-REC-BT.2020

  16. Kalantari, N.K., Ramamoorthi, R.: Deep high dynamic range imaging of dynamic scenes. ACM Trans. Graph. 36(4), 144 (2017)

    Article  Google Scholar 

  17. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

    Google Scholar 

  18. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the IEEE International Conference on Learning Representations (2015)

    Google Scholar 

  19. Kovaleski, R.P., Oliveira, M.M.: High-quality reverse tone mapping for a wide range of exposures. In: 2014 27th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pp. 49–56. IEEE (2014)

    Google Scholar 

  20. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)

    Google Scholar 

  21. Long, M., Cao, Z., Wang, J., Philip, S.Y.: Learning multiple tasks with multilinear relationship networks. In: Advances in Neural Information Processing Systems, pp. 1593–1602 (2017)

    Google Scholar 

  22. Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., Feris, R.: Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5334–5343 (2017)

    Google Scholar 

  23. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the IEEE International Conference on Computer Vision, vol. 2, pp. 416–423. IEEE (2001)

    Google Scholar 

  24. Masia, B., Serrano, A., Gutierrez, D.: Dynamic range expansion based on image statistics. Multimed. Tools Appl. 76(1), 631–648 (2017)

    Article  Google Scholar 

  25. Meylan, L., Daly, S., Süsstrunk, S.: The reproduction of specular highlights on high dynamic range displays. In: Color and Imaging Conference, vol. 1, pp. 333–338. Society for Imaging Science and Technology (2006)

    Google Scholar 

  26. Misra, I., Shrivastava, A., Gupta, A., Hebert, M.: Cross-stitch networks for multi-task learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994–4003 (2016)

    Google Scholar 

  27. Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic tone reproduction for digital images. ACM Trans. Graph. 21(3), 267–276 (2002)

    Article  Google Scholar 

  28. Rempel, A.G., et al.: Ldr2Hdr: on-the-fly reverse tone mapping of legacy video and photographs. ACM Trans. Graph. 26(3), 39. ACM (2007)

    Article  Google Scholar 

  29. Schubert, F., Schertler, K., Mikolajczyk, K.: A hands-on approach to high-dynamic-range and superresolution fusion. In: 2009 Workshop on Applications of Computer Vision (WACV), pp. 1–8. IEEE (2009)

    Google Scholar 

  30. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

    Google Scholar 

  31. SMPTE: High dynamic range electro-optical transfer function of mastering reference displays. SMPTE ST2084:2014 (2014)

    Google Scholar 

  32. Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 111–126. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16817-3_8

    Chapter  Google Scholar 

  33. Traonmilin, Y., Aguerrebere, C.: Simultaneous high dynamic range and superresolution imaging without regularization. SIAM J. Imaging Sci. 7(3), 1624–1644 (2014)

    Article  MathSciNet  Google Scholar 

  34. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)

    Article  MathSciNet  Google Scholar 

  35. Zhang, J., Lalonde, J.F.: Learning high dynamic range from outdoor panoramas. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4529–4538. IEEE (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Korea Advanced Institute of Science and Technology, Daejeon, Korea

    Soo Ye Kim & Munchurl Kim

Authors
  1. Soo Ye Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Munchurl Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Munchurl Kim .

Editor information

Editors and Affiliations

  1. IIIT Hyderabad, Hyderabad, India

    C. V. Jawahar

  2. ANU, Canberra, ACT, Australia

    Hongdong Li

  3. Simon Fraser University, Burnaby, BC, Canada

    Greg Mori

  4. ETH Zurich, Zurich, Zürich, Switzerland

    Konrad Schindler

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kim, S.Y., Kim, M. (2019). A Multi-purpose Convolutional Neural Network for Simultaneous Super-Resolution and High Dynamic Range Image Reconstruction. In: Jawahar, C., Li, H., Mori, G., Schindler, K. (eds) Computer Vision – ACCV 2018. ACCV 2018. Lecture Notes in Computer Science(), vol 11363. Springer, Cham. https://doi.org/10.1007/978-3-030-20893-6_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-20893-6_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-20892-9

  • Online ISBN: 978-3-030-20893-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation