Abstract
Binocular tone mapping is studied in the previous works to generate a fusible pair of LDR images in order to convey more visual content than one single LDR image. However, the existing methods are all based on monocular tone mapping operators. It greatly restricts the preservation of local details and global contrast in a binocular LDR pair. In this paper, we proposed the first binocular tone mapping operator to more effectively distribute visual content to an LDR pair, leveraging the great representability and interpretability of deep convolutional neural network. Based on the existing binocular perception models, novel loss functions are also proposed to optimize the output pairs in terms of local details, global contrast, content distribution, and binocular fusibility. Our method is validated with a qualitative and quantitative evaluation, as well as a user study. Statistics show that our method outperforms the state-of-the-art binocular tone mapping frameworks in terms of both visual quality and time performance.







Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Aubry, M., Paris, S., Hasinoff, S.W., Kautz, J., Durand, F.: Fast local laplacian filters: theory and applications. ACM Trans. Graph. (TOG) 33(5), 167 (2014)
Banterle, F., Artusi, A., Debattista, K., Chalmers, A.: Advanced High Dynamic Range Imaging. Taylor & Francis, CRC Press (2017)
Curtis, D.W., Rule, S.J.: Binocular processing of brightness information: a vector-sum model. J. Exp. Psychol. Human Percept. Perform. 4(1), 132 (1978)
Daly, S.J.: Visible differences predictor: an algorithm for the assessment of image fidelity. In: Human Vision, Visual Processing, and Digital Display III, vol. 1666, pp. 2–16. International Society for Optics and Photonics (1992). https://doi.org/10.1117/12.135952
De Weert, C.M., Levelt, W.J.M.: Binocular brightness combinations: additive and nonadditive aspects. Percept. Psychophys. 15(3), 551–562 (1974)
Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: ACM SIGGRAPH, pp. 369–378 (1997). https://doi.org/10.1145/258734.258884
Drago, F., Myszkowski, K., Annen, T., Chiba, N.: Adaptive logarithmic mapping for displaying high contrast scenes. In: Computer Graphics Forum, vol. 22, pp. 419–426. Wiley Online Library (2003). https://doi.org/10.1111/1467-8659.00689
Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. In: ACM Transactions on Graphics (TOG), vol. 21, pp. 257–266. ACM (2002). https://doi.org/10.1145/566570.566574
Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R.K., Unger, J.: Hdr image reconstruction from a single exposure using deep cnns. arXiv preprint arXiv:1710.07480 (2017)
Endo, Y., Kanamori, Y., Mitani, J.: Deep reverse tone mapping. ACM Trans. Graph. (TOG) 36(6), 177 (2017)
Engel, G.: The autocorrelation function and binocular brightness mixing. Vis. Res. 9(9), 1111–1130 (1969)
Fairchild, M.D.: The hdr photographic survey. In: Color and Imaging Conference, vol. 2007, pp. 233–238. Society for Imaging Science and Technology (2007)
Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. In: ACM Transactions on Graphics (TOG), ACM, vol. 27, p. 67 (2008). https://doi.org/10.1145/1399504.1360666
Feng, M., Loew, M.H.: Video-level binocular tone-mapping framework based on temporal coherency algorithm. In: 2017 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), IEEE, pp. 1–5 (2017)
Gharbi, M., Chen, J., Barron, J.T., Hasinoff, S.W., Durand, F.: Deep bilateral learning for real-time image enhancement. ACM Trans. Graph. (TOG) 36(4), 118 (2017)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014). https://papers.nips.cc/paper/5423-generative-adversarial-nets
Hasinoff, S.W., Sharlet, D., Geiss, R., Adams, A., Barron, J.T., Kainz, F., Chen, J., Levoy, M.: Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph. (TOG) 35(6), 192 (2016)
Hau Chua, S., Zhang, H., Hammad, M., Zhao, S., Goyal, S., Singh, K.: Colorbless: augmenting visual information for colorblind people with binocular luster effect. ACM Trans. Comput.-Hum. Interact. (TOCHI) 21(6), 32 (2015)
Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. (TOG) 35(4), 110 (2016)
Jähne, B., Haussecker, H., Geissler, P.: Handbook of computer vision and applications, vol. 2. Citeseer (1999)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) European Conference on Computer Vision, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, vol. 2, p. 4 (2017)
Legge, G.E.: Binocular contrast summation-II. Quadratic Summ. Vis. Res. 24(4), 385–394 (1984)
Legge, G.E., Rubin, G.S.: Binocular interactions in suprathreshold contrast perception. Percept. Psychophys. 30(1), 49–61 (1981)
Levelt, W.J.: Binocular brightness averaging and contour information. Br. J. Psychol. 56(1), 1–13 (1965)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)
Maehara, G., Goryo, K.: Binocular, monocular and dichoptic pattern masking. Opt. Rev. 12(2), 76–82 (2005)
Meese, T.S., Georgeson, M.A., Baker, D.H.: Interocular masking and summation indicate two stages of divisive contrast gain control. In: Twenty-Eighth European Conference on Visual Perception (2005)
Meese, T.S., Georgeson, M.A., Baker, D.H.: Binocular contrast vision at and above threshold. J. Vis. 6(11), 7–7 (2006)
Meese, T.S., Hess, R.F.: Low spatial frequencies are suppressively masked across spatial scale, orientation, field position, and eye of origin. J. Vis. 4(10), 2–2 (2004)
Nemoto, H., Korshunov, P., Hanhart, P., Ebrahimi, T.: Visual attention in ldr and hdr images. In: 9th International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM), EPFL-CONF-203873 (2015)
Paris, S., Hasinoff, S.W., Kautz, J.: Local laplacian filters: edge-aware image processing with a laplacian pyramid. ACM Trans. Graph. (TOG) 30(4), 1–68 (2011)
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch. In: NIPS-W (2017)
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016). https://doi.org/10.1109/CVPR.2016.278
Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., Myszkowski, K.: High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting. Morgan Kaufmann, Burlington (2010)
Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic tone reproduction for digital images. ACM Trans. Graph. (TOG) 21(3), 267–276 (2002)
Sajjadi, M.S.M., Schölkopf, B., Hirsch, M.: Enhancenet: Single image super-resolution through automated texture synthesis. In: IEEE International Conference on Computer Vision (ICCV) (2017)
Schlick, C.: Quantization techniques for visualization of high dynamic range pictures. In: Photorealistic Rendering Techniques, pp. 7–20. Springer (1995)
Sendik, O., Cohen-Or, D.: Deep correlations for texture synthesis. ACM Trans. Graph. (TOG) 36(5), 161 (2017). https://doi.org/10.1145/3015461
Smith, K., Krawczyk, G., Myszkowski, K., Seidel, H.P.: Beyond tone mapping: enhanced depiction of tone mapped hdr images. In: Computer Graphics Forum, vol. 25, pp. 427–438. Wiley Online Library (2006)
Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: AAAI, vol. 4, p. 12 (2017). https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/viewPaper/14806
Tumblin, J., Rushmeier, H.: Tone reproduction for realistic images. IEEE Comput. Graph. Appl. 13(6), 42–48 (1993). https://doi.org/10.1109/38.252554
Tumblin, J., Turk, G.: Lcis: A boundary hierarchy for detail-preserving contrast reduction. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 83–90. ACM Press/Addison-Wesley Publishing Co. (1999)
von Helmholtz, H., Southall, J.P.C.: Treatise on Physiological Optics, vol. 3. Courier Corporation, North Chelmsford (2005)
Wilson, H.R.: Binocular contrast, stereopsis, and rivalry: toward a dynamical synthesis. Vis. Res. 140, 89–95 (2017)
Ward, G.: A contrast-based scalefactor for luminance display. Graph. Gems IV, 415–421 (1994)
Xie, S., Tu, Z.: Holistically-nested edge detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1395–1403 (2015)
Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. arXiv preprint arXiv:1611.09969 (2016)
Yang, X., Zhang, L., Wong, T.T., Heng, P.A.: Binocular tone mapping. ACM Trans. Graph. (SIGGRAPH 2012 issue) 31(4), 93:1–93:10 (2012)
Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv preprint: arXiv:1511.07122 (2015)
Zhang, L., Zhang, L., Mou, X., Zhang, D., et al.: Fsim: a feature similarity index for image quality assessment. IEEE Trans. Image Process. 20(8), 2378–2386 (2011)
Zhang, Z., Hu, X., Liu, X., Wong, T.T.: Binocular tone mapping with improved overall contrast and local details. Comput. Graph. Forum 37(7), 433–442 (2018)
Acknowledgements
This project is supported by the Research Grants Council of the Hong Kong Special Administrative Region, under RGC General Research Fund (Project No. CUHK 14201017), and Shenzhen Science and Technology Programs (No. JCYJ20160429190300857, No. JCYJ20180507182410327, and No. JCYJ20180507182415428).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhang, Z., Han, C., He, S. et al. Deep binocular tone mapping. Vis Comput 35, 997–1011 (2019). https://doi.org/10.1007/s00371-019-01669-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-019-01669-8