Abstract
Illumination harmonization is an important problem for high-quality image composite. Given the source image and the target background, it aims to transform the foreground appearance that it looks in the same lighting condition as the target background. Because the ground-truth composite image is difficult to get, previous works can use only synthetic datasets, which however, provide with only artificially adjusted and limited inputs. In this paper we contribute to this problem in two aspects: 1) We introduce a semi-automatic approach to capture the ground-truth composite in real scenes, and then create a dataset that enables faithful evaluation of image harmonization methods. 2) We propose a simple yet effective harmonization method, namely the Gray Mean Scale (GMS), which models the foreground appearance transformation as channel-wise scales, and estimates the scales based on gray pixels of the source and the target background images. In experiments we evaluated the proposed method and compared it with previous methods, using both our dataset and previous synthetic datasets. A new benchmark thus is established for illumination harmonization in real environments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_3
Bychkovsky, V., Paris, S., Chan, E., Durand, F.: Learning photographic global tonal adjustment with a database of input/output image pairs. In: CVPR, pp. 97–104. IEEE (2011)
Calonder, M., Lepetit, V., Strecha, C., Fua, P.: BRIEF: binary robust independent elementary features. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 778–792. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_56
Cong, W., et al.: Image harmonization datasets: HCOCO, HAdobe5k, HFlickr, and Hday2night. arXiv preprint arXiv:1908.10526 (2019)
Finlayson, G.D., Zakizadeh, R., Gijsenij, A.: The reproduction angular error for evaluating the performance of illuminant estimation algorithms. IEEE TPAMI 39(7), 1482–1488 (2016)
Garon, M., Sunkavalli, K., Hadap, S., Carr, N., Lalonde, J.F.: Fast spatially-varying indoor lighting estimation. In: CVPR, pp. 6908–6917 (2019)
Lalonde, J.F., Efros, A.A.: Using color compatibility for assessing image realism. In: ICCV, pp. 1–8. IEEE (2007)
Li, Y., Liu, M.-Y., Li, X., Yang, M.-H., Kautz, J.: A closed-form solution to photorealistic image stylization. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 468–483. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_28
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Maloney, L.T., Wandell, B.A.: Color constancy: a method for recovering surface spectral reflectance. JOSA A 3(1), 29–33 (1986)
P’erez, P., Gangnet, M., Blake, A.: Poisson image editing. In: ACM SIGGRAPH, pp. 313–318 (2003)
Qian, Y., Kamarainen, J.K., Nikkanen, J., Matas, J.: On finding gray pixels. In: CVPR, pp. 8062–8070 (2019)
Reinhard, E., Adhikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. IEEE Comput. Graphics Appl. 21(5), 34–41 (2001)
Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: LabelMe: a database and web-based tool for image annotation. Int. J. Comput. Vision 77(1–3), 157–173 (2008). https://doi.org/10.1007/s11263-007-0090-8
Shafer, S.A.: Using color to separate reflection components. Color Res. Appl. 10(4), 210–218 (1985)
Shi, L.: Re-processed version of the gehler color constancy dataset of 568 images (2000). http://www.cs.sfu.ca/color/data/
Tsai, Y.H., Shen, X., Lin, Z., Sunkavalli, K., Lu, X., Yang, M.H.: Deep image harmonization. In: CVPR, pp. 3789–3797 (2017)
Wu, H., Zheng, S., Zhang, J., Huang, K.: GP-GAN: towards realistic high-resolution image blending. In: ACM International Conference on Multimedia, pp. 2487–2495 (2019)
Xing, G., Liu, Y., Ling, H., Granier, X., Zhang, Y.: Automatic spatially varying illumination recovery of indoor scenes based on a single RGB-D image. IEEE TVCG, 1–14 (2018)
Xue, S., Agarwala, A., Dorsey, J., Rushmeier, H.: Understanding and improving the realism of image composites. ACM Trans. Graph. (TOG) 31(4), 84 (2012)
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution, pp. 4471–4480 (2019)
Zhu, J.Y., Krahenbuhl, P., Shechtman, E., Efros, A.A.: Learning a discriminative model for the perception of realism in composite images. In: ICCV, pp. 3943–3951 (2015)
Acknowledgements
This work is supported by Industrial Internet Innovation and Development Project in 2019 of China, NSF of China (No. 61772318).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Song, S., Zhong, F., Qin, X., Tu, C. (2020). Illumination Harmonization with Gray Mean Scale. In: Magnenat-Thalmann, N., et al. Advances in Computer Graphics. CGI 2020. Lecture Notes in Computer Science(), vol 12221. Springer, Cham. https://doi.org/10.1007/978-3-030-61864-3_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-61864-3_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61863-6
Online ISBN: 978-3-030-61864-3
eBook Packages: Computer ScienceComputer Science (R0)