research-article

Hybrid Conditional Deep Inverse Tone Mapping

Authors:

Xianming LiuAuthors Info & Claims

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

Pages 1016 - 1024

https://doi.org/10.1145/3503161.3548129

Published: 10 October 2022 Publication History

Abstract

Emerging modern displays are capable to render ultra-high definition (UHD) media contents with high dynamic range (HDR) and wide color gamut (WCG). Although more and more native contents as such have been getting produced, the total amount is still in severe lack. Considering the massive amount of legacy contents with standard dynamic range (SDR) which may be exploitable, the urgent demand for proper conversion techniques thus springs up. In this paper, we try to tackle the conversion task from SDR to HDR-WCG for media contents and consumer displays. We propose a deep learning based SDR-to-HDR solution, Hybrid Conditional Deep Inverse Tone Mapping (HyCondITM), which is an end-to-end trainable framework including global transform, local adjustment, and detail refinement in a single unified pipeline. We present a hybrid condition network that can simultaneously extract both global and local priors for guidance to achieve scene-adaptive and spatially-variant manipulations. Experiments show that our method achieves state-of-the-art performance in both quantitative comparisons and visual quality, out-performing the previous methods.

References

[1]

Ahmet Oguz Akyüz, Roland Fleming, Bernhard E. Riecke, Erik Reinhard, and Heinrich H. Bülthoff. 2007. Do HDR Displays Support LDR Content? A Psychophysical Evaluation. In ACM SIGGRAPH 2007 Papers (SIGGRAPH '07). Association for Computing Machinery, New York, NY, USA, 38--es.

[2]

Tunç O. Aydin, Rafal Mantiuk, and Hans-Peter Seidel. 2008. Extending Quality Metrics to Full Luminance Range Images. In Human Vision and Electronic Imaging XIII, Vol. 6806. International Society for Optics and Photonics, San Jose, California, United States, 68060B.

[3]

Francesco Banterle, Alessandro Artusi, Kurt Debattista, and Alan Chalmers. 2011. Advanced High Dynamic Range Imaging: Theory and Practice. AK Peters (CRC Press), Natick, MA, USA.

Digital Library

[4]

Francesco Banterle, Patrick Ledda, Kurt Debattista, and Alan Chalmers. 2006. Inverse Tone Mapping. In Proceedings of the 4th International Conference on Computer Graphics and Interactive Techniques in Australasia and Southeast Asia (GRAPHITE '06). Association for Computing Machinery, New York, NY, USA, 349--356.

Digital Library

[5]

Tim Borer and Andrew Cotton. 2016. A Display-Independent High Dynamic Range Television System. SMPTE Motion Imaging Journal 125, 4 (2016), 50--56.

[6]

Guannan Chen, Lijie Zhang, Mengdi Sun, Yan Gao, Pablo Navarrete Michelini, and YanHong Wu. 2021. Single-Image HDR Reconstruction with Task-specific Network Based on Channel Adaptive RDN. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, Nashville, TN, USA, 398--403.

[7]

Xiangyu Chen, Yihao Liu, Zhengwen Zhang, Yu Qiao, and Chao Dong. 2021. HDRUNet: Single Image HDR Reconstruction with Denoising and Dequantization. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, Nashville, TN, USA, 354--363.

[8]

Xiangyu Chen, Zhengwen Zhang, Jimmy S. Ren, Lynhoo Tian, Yu Qiao, and Chao Dong. 2021. A New Journey from SDRTV to HDRTV. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Montreal, QC, Canada, 4480--4489.

[9]

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a Deep Convolutional Network for Image Super-Resolution. In Computer Vision -- ECCV 2014 (Lecture Notes in Computer Science), David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 184--199.

[10]

Gabriel Eilertsen, Saghi Hajisharif, Param Hanji, Apostolia Tsirikoglou, Rafal K. Mantiuk, and Jonas Unger. 2021. How to Cheat with Metrics in Single-Image HDR Reconstruction. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). IEEE, Montreal, BC, Canada, 3981--3990.

[11]

Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafal K. Mantiuk, and Jonas Unger. 2017. HDR Image Reconstruction from a Single Exposure Using Deep CNNs. ACM Transactions on Graphics 36, 6 (Nov. 2017), 178:1--178:15.

Digital Library

[12]

Yuki Endo, Yoshihiro Kanamori, and Jun Mitani. 2017. Deep Reverse Tone Mapping. ACM Transactions on Graphics 36, 6 (Nov. 2017), 177:1--177:10.

Digital Library

[13]

Michaël Gharbi, Jiawen Chen, Jonathan T. Barron, SamuelW. Hasinoff, and Frédo Durand. 2017. Deep Bilateral Learning for Real-Time Image Enhancement. ACM Transactions on Graphics 36, 4 (July 2017), 118:1--118:12.

Digital Library

[14]

Jingwen He, Yihao Liu, Yu Qiao, and Chao Dong. 2020. Conditional Sequential Modulation for Efficient Global Image Retouching. In Computer Vision -- ECCV 2020 (Lecture Notes in Computer Science), Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 679--695.

[15]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Las Vegas, NV, USA, 770--778.

[16]

G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the Dimensionality of Data with Neural Networks. Science 313, 5786 (July 2006), 504--507.

[17]

Yongqing Huo, Fan Yang, Le Dong, and Vincent Brost. 2014. Physiological Inverse Tone Mapping Based on Retina Response. The Visual Computer 30, 5 (May 2014), 507--517.

[18]

ITU-R. 2015. Colour Conversion from Recommendation ITU-R BT.709 to Recommendation ITU-R BT.2020. Technical Report Rec. ITU-R BT.2087-0. ITU-R.

[19]

ITU-R. 2015. Parameter Values for Ultra-High Definition Television Systems for Production and International Programme Exchange. Technical Report Rec. ITU-R BT.2020--2. ITU-R.

[20]

ITU-R. 2018. Image Parameter Values for High Dynamic Range Television for Use in Production and International Programme Exchange. Technical Report Rec. ITU-R BT.2100--2. ITU-R.

[21]

ITU-R. 2019. Objective Metric for the Assessment of the Potential Visibility of Colour Differences in Television. Technical Report Rec. ITU-R BT.2124-0. ITU-R.

[22]

Soo Ye Kim, Dae-Eun Kim, and Munchurl Kim. 2019. ITM-CNN: Learning the Inverse Tone Mapping from Low Dynamic Range Video to High Dynamic Range Displays Using Convolutional Neural Networks. In Computer Vision -- ACCV 2018 (Lecture Notes in Computer Science), C. V. Jawahar, Hongdong Li, Greg Mori, and Konrad Schindler (Eds.). Springer International Publishing, Cham, 395--409.

[23]

S. Y. Kim, J. Oh, and M. Kim. 2019. Deep SR-ITM: Joint Learning of Super- Resolution and Inverse Tone-Mapping for 4K UHD HDR Applications. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Seoul, Korea (South), 3116--3125.

[24]

Soo Ye Kim, Jihyong Oh, and Munchurl Kim. 2020. JSI-GAN: GAN-Based Joint Super-Resolution and Inverse Tone-Mapping with Pixel-Wise Task-Specific Filters for UHD HDR Video. Proceedings of the AAAI Conference on Artificial Intelligence 34, 07 (April 2020), 11287--11295.

[25]

Rafael P. Kovaleski and Manuel M. Oliveira. 2014. High-Quality Reverse Tone Mapping for a Wide Range of Exposures. In 2014 27th SIBGRAPI Conference on Graphics, Patterns and Images. IEEE, Rio de Janeiro, Brazil, 49--56.

[26]

Siyeong Lee, Gwon Hwan An, and Suk-Ju Kang. 2018. Deep Recursive HDRI: Inverse Tone Mapping Using Generative Adversarial Networks. In Computer Vision -- ECCV 2018 (Lecture Notes in Computer Science), Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer International Publishing, Cham, 613--628.

[27]

Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. 2020. Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Seattle, WA, USA, 1648-- 1657.

[28]

Rafal Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in All Luminance Conditions. ACM Transactions on Graphics 30, 4 (July 2011), 40:1--40:14.

Digital Library

[29]

D. Marnerides, T. Bashford-Rogers, J. Hatchett, and K. Debattista. 2018. Expand- Net: A Deep Convolutional Neural Network for High Dynamic Range Expansion from Low Dynamic Range Content. Computer Graphics Forum 37, 2 (2018), 37--49.

[30]

Augustus Odena, Vincent Dumoulin, and Chris Olah. 2016. Deconvolution and Checkerboard Artifacts. Distill 1, 10 (Oct. 2016), e3.

[31]

Eduardo Pérez-Pellitero, Sibi Catley-Chandar, Ale? Leonardis, Radu Timofte, Xian Wang, Yong Li, TaoWang, Fenglong Song, Zhen Liu,Wenjie Lin, Xinpeng Li, Qing Rao, Ting Jiang, Mingyan Han, Haoqiang Fan, Jian Sun, Shuaicheng Liu, Xiangyu Chen, Yihao Liu, Zhengwen Zhang, Yu Qiao, Chao Dong, Evelyn Yi Lyn Chee, Shanlan Shen, Yubo Duan, Guannan Chen, Mengdi Sun, Yan Gao, Lijie Zhang, Akhil K A, Jiji C V, S M A Sharif, Rizwan Ali Naqvi, Mithun Biswas, Sungjun Kim, Chenjie Xia, Bowen Zhao, Zhangyu Ye, Xiwen Lu, Yanpeng Cao, Jiangxin Yang, Yanlong Cao, Green Rosh K S, Sachin Deepak Lomte, Nikhil Krishnan, and B H Pawan Prasad. 2021. NTIRE 2021 Challenge on High Dynamic Range Imaging: Dataset, Methods and Results. In 2021 IEEE/CVF Conference on Computer Vision and Pattern RecognitionWorkshops (CVPRW). IEEE, Nashville, TN, USA, 691--700.

[32]

Allan G. Rempel, Matthew Trentacoste, Helge Seetzen, H. David Young,Wolfgang Heidrich, Lorne Whitehead, and Greg Ward. 2007. Ldr2Hdr: On-the-Fly Reverse Tone Mapping of Legacy Video and Photographs. ACM Transactions on Graphics 26, 3 (July 2007), 39--es.

Digital Library

[33]

Marcel Santana Santos, Tsang Ing Ren, and Nima Khademi Kalantari. 2020. Single Image HDR Reconstruction Using a CNN with Masked Features and Perceptual Loss. ACM Transactions on Graphics 39, 4 (July 2020), 80:80:1--80:80:10.

Digital Library

[34]

Tao Wang, Yong Li, Jingyang Peng, Yipeng Ma, Xian Wang, Fenglong Song, and Youliang Yan. 2021. Real-Time Image Enhancer via Learnable Spatial-aware 3D Lookup Tables. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Montreal, QC, Canada, 2451--2460.

[35]

Xintao Wang, Kelvin C.K. Chan, Ke Yu, Chao Dong, and Chen Change Loy. 2019. EDVR: Video Restoration With Enhanced Deformable Convolutional Networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, Long Beach, CA, USA, 1954--1963.

[36]

Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. 2018. Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Salt Lake City, UT, USA, 606--615.

[37]

Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. 2018. CBAM: Convolutional Block Attention Module. In Computer Vision -- ECCV 2018 (Lecture Notes in Computer Science), Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer International Publishing, Cham, 3--19.

[38]

Zhicheng Yan, Hao Zhang, Baoyuan Wang, Sylvain Paris, and Yizhou Yu. 2016. Automatic Photo Adjustment Using Deep Neural Networks. ACM Transactions on Graphics 35, 2 (Feb. 2016), 11:1--11:15.

Digital Library

[39]

Hui Zeng, Jianrui Cai, Lida Li, Zisheng Cao, and Lei Zhang. 2022. Learning Image-Adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-Time. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 4 (April 2022), 2058--2073.

[40]

Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. 2017. Loss Functions for Image Restoration With Neural Networks. IEEE Transactions on Computational Imaging 3, 1 (March 2017), 47--57.

Cited By

Wang ZHe G(2024)Multi-Frame Content-Aware Mapping Network for Standard-Dynamic-Range to High-Dynamic-Range Television Artifact RemovalSensors10.3390/s2401029924:1(299)Online publication date: 4-Jan-2024
https://doi.org/10.3390/s24010299
He GWang SXu KAlameda-Pineda XJin QOria VToni L(2024)High-dynamic Range Video Generation Method Based On Adaptive Priors Dynamic MappingProceedings of the 2nd International Workshop on Methodologies for Multimedia10.1145/3689089.3689703(2-9)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3689089.3689703
Nam YKim JShim JKang S(2024)Deep Conditional HDRI: Inverse Tone Mapping via Dual Encoder-Decoder Conditioning MethodIEEE Transactions on Multimedia10.1109/TMM.2024.337989026(8504-8515)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3379890
Show More Cited By

Index Terms

Hybrid Conditional Deep Inverse Tone Mapping
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Image processing
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Video Inverse Tone Mapping Network with Luma and Chroma Mapping
MM '23: Proceedings of the 31st ACM International Conference on Multimedia

\beginabstract With the popularity of consumer high dynamic range (HDR) display devices, video inverse tone mapping (iTM) has become a research hotspot. However, existing methods are designed based on a perceptual non-uniformity color space (e.g., RGB ...
A framework for inverse tone mapping
Abstract
In recent years many tone mapping operators (TMOs) have been presented in order to display high dynamic range images (HDRI) on typical display devices. TMOs compress the luminance range while trying to maintain contrast. The inverse of tone ...
Tone-mapping high dynamic range images by novel histogram adjustment

In this paper, we present novel histogram adjustment methods for displaying high dynamic range image. We first present a global histogram adjustment based tone mapping operator, which well reproduces global contrast for high dynamic range images. We ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

October 2022

7537 pages

ISBN:9781450392037

DOI:10.1145/3503161

General Chairs:
João Magalhães
NOVA University of Lisbon, Portugal
,
Alberto del Bimbo
University of Florence, Italy
,
Shin'ichi Satoh
National Institute of Informatics, Japan
,
Nicu Sebe
University of Trento, Italy
,
Program Chairs:
Xavier Alameda-Pineda
Inria, Grenoble, France
,
Qin Jin
Renmin University of China, China
,
Vincent Oria
New Jersey Institute of Technology, USA
,
Laura Toni
University College London, UK

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Conference

MM '22

Sponsor:

SIGMM

MM '22: The 30th ACM International Conference on Multimedia

October 10 - 14, 2022

Lisboa, Portugal

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
290
Total Downloads

Downloads (Last 12 months)76
Downloads (Last 6 weeks)10

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang ZHe G(2024)Multi-Frame Content-Aware Mapping Network for Standard-Dynamic-Range to High-Dynamic-Range Television Artifact RemovalSensors10.3390/s2401029924:1(299)Online publication date: 4-Jan-2024
https://doi.org/10.3390/s24010299
He GWang SXu KAlameda-Pineda XJin QOria VToni L(2024)High-dynamic Range Video Generation Method Based On Adaptive Priors Dynamic MappingProceedings of the 2nd International Workshop on Methodologies for Multimedia10.1145/3689089.3689703(2-9)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3689089.3689703
Nam YKim JShim JKang S(2024)Deep Conditional HDRI: Inverse Tone Mapping via Dual Encoder-Decoder Conditioning MethodIEEE Transactions on Multimedia10.1109/TMM.2024.337989026(8504-8515)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3379890
Zhou FYuan SLiang ZDuan JQiu G(2024)A Dataset and Model for the Visual Quality Assessment of Inversely Tone-Mapped HDR VideosIEEE Transactions on Image Processing10.1109/TIP.2023.334309933(366-381)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2023.3343099
Guo CFan LZhang QLiu HLiu KJiang X(2023)Redistributing the Precision and Content in 3D-LUT-based Inverse Tone-mapping for HDR/WCG DisplayProceedings of the 20th ACM SIGGRAPH European Conference on Visual Media Production10.1145/3626495.3626503(1-10)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3626495.3626503
Guo CFan LXue ZJiang X(2023)Learning a Practical SDR-to-HDRTV Up-conversion using New Dataset and Degradation Models2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.02129(22231-22241)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.02129
He GLong SXu LWu CYu WZhou J(2023)Global priors guided modulation network for joint super-resolution and SDRTV-to-HDRTVNeurocomputing10.1016/j.neucom.2023.126590554:COnline publication date: 14-Oct-2023
https://dl.acm.org/doi/10.1016/j.neucom.2023.126590

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten