research-article

Temporally coherent local tone mapping of HDR video

Authors:

Tunç Ozan Aydin,

Nikolce Stefanoski,

Aljoscha SmolicAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 33, Issue 6

Article No.: 196, Pages 1 - 13

https://doi.org/10.1145/2661229.2661268

Published: 19 November 2014 Publication History

Abstract

Recent subjective studies showed that current tone mapping operators either produce disturbing temporal artifacts, or are limited in their local contrast reproduction capability. We address both of these issues and present an HDR video tone mapping operator that can greatly reduce the input dynamic range, while at the same time preserving scene details without causing significant visual artifacts. To achieve this, we revisit the commonly used spatial base-detail layer decomposition and extend it to the temporal domain. We achieve high quality spatiotemporal edge-aware filtering efficiently by using a mathematically justified iterative approach that approximates a global solution. Comparison with the state-of-the-art, both qualitatively, and quantitatively through a controlled subjective experiment, clearly shows our method's advantages over previous work. We present local tone mapping results on challenging high resolution scenes with complex motion and varying illumination. We also demonstrate our method's capability of preserving scene details at user adjustable scales, and its advantages for low light video sequences with significant camera noise.

Supplementary Material

ZIP File (a196.zip)

Supplemental material.

Download
705.28 MB

References

[1]

adams, A. 1981. The Print, The Ansel Adams Photography Series 3. New York Graphic Society.

[2]

Aubry, M., Paris, S., Hasinoff, S. Kautz, J., and Durand, F. 2014. Fast local laplacian filters: Theory and applications. ACM Trans. Graph. (To Appear) 33, 4.

Digital Library

[3]

Bennett, E. P., and McMillan, L. 2005. Video enhancement using per-pixel virtual exposures. ACM Trans. Graph. 24, 3.

Digital Library

[4]

Benoit, A., Alleysson, D., Herault, J., and Callet, P. L. 2009. Spatiotemporal tone mapping operator based on a retina model. In CCIW, Springer, vol. 5646 of Lecture Notes in Computer Science, 12--22.

Digital Library

[5]

Boitard, R., Bouatouch, K., Cozot, R., Thoreau, D., and Gruson, A. 2012. Temporal coherency for video tone mapping. Proc. SPIE 8499.

[6]

Boitard, R., Cozot, R., Thoreau, D., and Bouatouch, K. 2014. Survey of Temporal Brightness Artifacts in Video Tone Mapping. In HDRi2014.

[7]

Boitard, R., Cozot, R., Thoreau, D., and Bouatouch, K. 2014. Zonal brightness coherency for video tone mapping. Signal Processing: Image Communication 29, 2, 229--246.

Digital Library

[8]

Cigla, C., and Alatan, A. A. 2013. Information permeability for stereo matching. Signal Processing: Image Communication 28, 9, 1072--1088.

Digital Library

[9]

Drago, F., Myszkowski, K., Annen, T., and Chiba, N. 2003. Adaptive logarithmic mapping for displaying high contrast scenes. Computer Graphics Forum 22, 3, 419--426.

[10]

Durand, F., and Dorsey, J. 2002. Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21, 3, 257--266.

Digital Library

[11]

Eilertsen, G., Wanat, R., Mantiuk, R. K., and Unger, J. 2013. Evaluation of tone mapping operators for hdr-video. Computer Graphics Forum 32, 7, 275--284.

[12]

Farbman, Z., Fattal, R., Lischinski, D., and Szeliski, R. 2008. Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph. 27, 3, 67:1--67:10.

Digital Library

[13]

Fattal, R., Lischinski, D., and Werman, M. 2002. Gradient domain high dynamic range compression. ACM Trans. Graph. 21, 3, 249--256.

Digital Library

[14]

Fattal, R. 2009. Edge-avoiding wavelets and their applications. ACM Trans. Graph. 28, 3, 22:1--22:10.

Digital Library

[15]

Ferwerda, J. A., Pattanaik, S. N., Shirley, P., and Greenberg, D. P. 1996. A model of visual adaptation for realistic image synthesis. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '96, 249--258.

Digital Library

[16]

Gastal, E. S. L., and Oliveira, M. M. 2011. Domain transform for edge-aware image and video processing. ACM Trans. Graph. 30, 4, 69:1--69:12.

Digital Library

[17]

Guthier, B., S., K., M., E., and W., E. 2011. Flicker reduction in tone mapped high dynamic range video. Proc. SPIE 7866.

[18]

He, K., Sun, J., and Tang, X. 2013. Guided image filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 6, 1397--1409.

Digital Library

[19]

Irawan, P., Ferwerda, J. A., and Marschner, S. R. 2005. Perceptually based tone mapping of high dynamic range image streams. In Proceedings of Eurographics Conf. on Rendering Techniques, EGSR'05, 231--242.

Digital Library

[20]

Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2003. High dynamic range video. ACM Trans. Graph. 22, 3, 319--325.

Digital Library

[21]

Kiser, C., Reinhard, E., Tocci, M., and Tocci, N. 2012. Real time automated tone mapping system for HDR video. In Proceedings of the IEEE International Conference on Image Processing, 2749--2752.

[22]

Kronander, J., Gustavson, S., Bonnet, G., and Unger, J. 2013. Unified HDR reconstruction from raw cfa data. In Computational Photography (ICCP), 2013 IEEE International Conference on, 1--9.

[23]

Kronander, J., Gustavson, S., Bonnet, G., Ynmerman, J., and Unger, J. 2013. A unified framework for multi-sensor HDR video reconstruction. In Signal Processing: Image Communications.

Digital Library

[24]

Lang, M., Wang, O., Aydin, T., Smolic, A., and Gross, M. 2012. Practical temporal consistency for image-based graphics applications. ACM Trans. Graph. 31, 4 (July), 34:1--34:8.

Digital Library

[25]

Ledda, P., Santos, L. P., and Chalmers, A. 2004. A local model of eye adaptation for high dynamic range images. AFRIGRAPH, 151--160.

Digital Library

[26]

Lee, C., and Kim, C.-S. 2007. Gradient domain tone mapping of high dynamic range videos. In Image Processing, 2007. ICIP 2007. IEEE International Conference on, vol. 3, III -- 461--III -- 464.

[27]

Lischinski, D., Farbman, Z., Uyttendaele, and M., Szeliski, R. 2006. Interactive local adjustment of tonal values. ACM Trans. Graph., 646--653.

Digital Library

[28]

Mac Aodha, O., Humayun, A., Pollefeys, M., and Brostow, G. J. 2013. Learning a confidence measure for optical flow. IEEE Trans. Pattern Anal. Mach. Intell. 35, 5, 1107--1120.

Digital Library

[29]

Mantiuk, R., Myszkowski, K., and Seidel, H.-P. 2006. A perceptual framework for contrast processing of high dynamic range images. ACM Trans. Appl. Percept. 3, 3, 286--308.

Digital Library

[30]

Mantiuk, R., Daly, S., and Kerofsky, L. 2008. Display adaptive tone mapping. ACM Trans. Graph. 27, 3, 68:1--68:10.

Digital Library

[31]

Meyer, C. D. 2001. Matrix Analysis and Applied Linear Algebra. SIAM: Society for Industrial and Applied Mathematics.

Digital Library

[32]

Milanfar, P. 2013. A tour of modern image filtering. IEEE Signal Processing Magazine 30, 1, 106--128.

[33]

Mrázek, P., Weickert, J., and Bruhn, A. 2004. On robust estimation and smoothing with spatial and tonal kernels. In Proc. Dagstuhl Seminar: Geometric Properties from Incomplete Data, Springer, 3--88722.

[34]

Nordstrom, K. N. 1989. Biased anisotropic diffusion--a unified regularization and diffusion approach to edge detection. Tech. rep., EECS Dept., UC Berkeley.

Digital Library

[35]

Paris, S., Hasinoff, S. W., and Kautz, J. 2011. Local laplacian filters: Edge-aware image processing with a laplacian pyramid. ACM Trans. Graph. 30, 4, 68:1--68:12.

Digital Library

[36]

Pattanaik, S. N., Tumblin, J., Yee, H., and Greenberg, D. P. 2000. Time-dependent visual adaptation for fast realistic image display. In Proc. of Conf. on Computer Graphics and Interactive Techniques, SIGGRAPH '00, 47--54.

Digital Library

[37]

Petit, J., and Mantiuk, R. K. 2013. Assessment of video tone-mapping: Are cameras s-shaped tone-curves good enough? Journal of Visual Communication and Image Representation 24, 7, 1020--1030.

Digital Library

[38]

Ramsey, S. D., Johnson III, J. T., and Hansen, C. 2004. Adaptive temporal tone mapping. In Proceedings of the 7th IASTED International Conference on Computer Graphics and Imaging, 124--128.

[39]

Reinhard, E., and Devlin, K. 2005. Dynamic range reduction inspired by photoreceptor physiology. IEEE Transactions on Visualization and Computer Graphics 11, 13--24.

Digital Library

[40]

Reinhard, E., Stark, M., Shirley, P., and Ferwerda, J. 2002. Photographic tone reproduction for digital images. ACM Trans. Graph. 21, 3, 267--276.

Digital Library

[41]

Reinhard, E., Ward, G., Pattanaik, S., Debevec, P., Heidrich, W., and Myszkowski, K. 2010. HDR Imaging - Acquisition, Display, and Image-Based Lighting, Second Edition. Morgan Kaufmann.

[42]

Reinhard, E., Pouli, T., Kunkel, T., Long, B., Ballestad, A., and Damberg, G. 2012. Calibrated image appearance reproduction. ACM Trans. Graph. 31, 6 (Nov.), 201:1--201:11.

Digital Library

[43]

Tocci, M. D., Kiser, C., Tocci, N., and Sen, P. 2011. A versatile hdr video production system. ACM Trans. Graph. 30, 4, 41:1--41:10.

Digital Library

[44]

Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In ICCV, 839--846.

Digital Library

[45]

Van Hateren, J. H. 2006. Encoding of high dynamic range video with a model of human cones. ACM Trans. Graph. 25, 4, 1380--1399.

Digital Library

[46]

Ye, G., Garces, E., Liu, Y., Dai, Q., and Gutierrez, D. 2014. Intrinsic Video and Applications. ACM Trans. Graph. (To Appear) 33, 4.

Digital Library

[47]

Zimmer, H., Bruhn, A., and Weickert, J. 2011. Optic Flow in Harmony. International Journal of Computer Vision 93, 3, 368--388.

Digital Library

Cited By

Huang XZhang QFeng YLi HWang Q(2024)LTM-NeRF: Embedding 3D Local Tone Mapping in HDR Neural Radiance FieldIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.344862046:12(10944-10959)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TPAMI.2024.3448620
Cao CYue HLiu XYang J(2024)Unsupervised HDR Image and Video Tone Mapping via Contrastive LearningIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.329035134:2(786-798)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/TCSVT.2023.3290351
Jiang YZhu JLi LMa H(2024)A Joint Network for Low-Light Image Enhancement Based on RetinexCognitive Computation10.1007/s12559-024-10347-4Online publication date: 16-Sep-2024
https://doi.org/10.1007/s12559-024-10347-4
Show More Cited By

Index Terms

Temporally coherent local tone mapping of HDR video
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Video segmentation
      2. Computer vision tasks
        Scene understanding
  2. Computer graphics
    1. Image manipulation
    2. Rendering

Recommendations

Real-time noise-aware tone mapping

Real-time high quality video tone mapping is needed for many applications, such as digital viewfinders in cameras, display algorithms which adapt to ambient light, in-camera processing, rendering engines for video games and video post-processing. We ...
Zonal brightness coherency for video tone mapping

Tone Mapping Operators (TMOs) compress High Dynamic Range (HDR) contents to address Low Dynamic Range (LDR) displays. While many solutions have been designed over the last decade, only few of them can cope with video sequences. Indeed, these TMOs tone ...
Video Inverse Tone Mapping Network with Luma and Chroma Mapping
MM '23: Proceedings of the 31st ACM International Conference on Multimedia

\beginabstract With the popularity of consumer high dynamic range (HDR) display devices, video inverse tone mapping (iTM) has become a research hotspot. However, existing methods are designed based on a perceptual non-uniformity color space (e.g., RGB ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 33, Issue 6

November 2014

704 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2661229

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 November 2014

Published in TOG Volume 33, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
1,177
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Huang XZhang QFeng YLi HWang Q(2024)LTM-NeRF: Embedding 3D Local Tone Mapping in HDR Neural Radiance FieldIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.344862046:12(10944-10959)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TPAMI.2024.3448620
Cao CYue HLiu XYang J(2024)Unsupervised HDR Image and Video Tone Mapping via Contrastive LearningIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.329035134:2(786-798)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/TCSVT.2023.3290351
Jiang YZhu JLi LMa H(2024)A Joint Network for Low-Light Image Enhancement Based on RetinexCognitive Computation10.1007/s12559-024-10347-4Online publication date: 16-Sep-2024
https://doi.org/10.1007/s12559-024-10347-4
Tariq TMatsuda NPenner EJia JLanman DNinan AChapiro A(2023)Perceptually Adaptive Real-Time Tone MappingSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618222(1-10)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.1145/3610548.3618222
Sheng BLi PAli RChen C(2022)Improving Video Temporal Consistency via Broad Learning SystemIEEE Transactions on Cybernetics10.1109/TCYB.2021.307931152:7(6662-6675)Online publication date: Jul-2022
https://doi.org/10.1109/TCYB.2021.3079311
Walbrecq CMarc IMagnier BLafon-Pham D(2021)Video logarithmic tone mapping operator with automatic calculation of the logarithmic function base2021 IEEE International Conference on Imaging Systems and Techniques (IST)10.1109/IST50367.2021.9651455(1-5)Online publication date: 24-Aug-2021
https://doi.org/10.1109/IST50367.2021.9651455
Eom CPark HHam B(2020)Temporally Consistent Depth Prediction With Flow-Guided Memory UnitsIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2019.294209621:11(4626-4636)Online publication date: Nov-2020
https://doi.org/10.1109/TITS.2019.2942096
Wang WYang SXu JLiu J(2020)Consistent Video Style Transfer via Relaxation and RegularizationIEEE Transactions on Image Processing10.1109/TIP.2020.302401829(9125-9139)Online publication date: 2020
https://doi.org/10.1109/TIP.2020.3024018
Huang HXu SCai JLiu WHu S(2020)Temporally Coherent Video Harmonization Using Adversarial NetworksIEEE Transactions on Image Processing10.1109/TIP.2019.292555029(214-224)Online publication date: 2020
https://doi.org/10.1109/TIP.2019.2925550
Niklaus SLiu F(2020)Softmax Splatting for Video Frame Interpolation2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR42600.2020.00548(5436-5445)Online publication date: Jun-2020
https://doi.org/10.1109/CVPR42600.2020.00548
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents