Abstract
Illumination is a very important environmental condition. Objects in different illumination environments will present different light and shadow effects. Different kinds of illumination sources will cause different brightness and colors on the surface of the object. The conversion of illumination in two pictures is an interesting and challenging new task, which will be useful in the fields of photography and computer graphics. To solve this problem, we propose a novel solution with three stages: illumination classification, One-to-One Relighting, and Any-to-Any Relighting. Our solution can accurately classify the illumination condition of the input image and can change the direction of the illumination source from any direction to another. We evaluate our methods on VIDIT, a rendered dataset of artificial scenes. The proposed solution produces good results under different light conditions.
L. Dong, Y. Zhu, Z. Jiang and X. He—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Mean Perceptual Score (MPS): the official evaluation protocol used in the AIM2020 relighting challenge. \(\text {MPS}=0.5\cdot (\text {SSIM}+(1-\text {LPIPS}))\).
References
Dherse, A.P., Everaert, M., Gwizdala, J.J.: Scene relighting with illumination estimation in the latent space on an encoder-decoder scheme. ArXiv abs/2006.02333 (2020)
El Helou, M., Zhou, R., Süsstrunk, S., Timofte, R., et al.: AIM 2020: scene relighting and illumination estimation challenge. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020 Workshops. LNCS, vol. 12537, pp. 499–518. Springer, Cham (2020)
Gafton, P., Maraz, E.: 2D image relighting with image-to-image translation. ArXiv abs/2006.07816 (2020)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, 7–13 December 2015, pp. 1026–1034 (2015). https://doi.org/10.1109/ICCV.2015.123
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
He, X., Cheng, K., Chen, Q., Hu, Q., Wang, P., Cheng, J.: Compact global descriptor for neural networks. CoRR abs/1907.09665 (2019). http://arxiv.org/abs/1907.09665
Helou, M.E., Zhou, R., Barthas, J., Süsstrunk, S.: VIDIT: virtual image dataset for illumination transfer. CoRR abs/2005.05460 (2020). https://arxiv.org/abs/2005.05460
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Huang, X., Liu, M.Y., Belongie, S.J., Kautz, J.: Multimodal unsupervised image-to-image translation. ArXiv abs/1804.04732 (2018)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976 (2017)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. ArXiv abs/1603.08155 (2016)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2015)
Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. ArXiv abs/1703.00848 (2017)
Liu, M., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA, 4–9 December 2017, pp. 700–708 (2017). http://papers.nips.cc/paper/6672-unsupervised-image-to-image-translation-networks
Luo, P., Ren, J., Peng, Z., Zhang, R., Li, J.: Differentiable learning-to-normalize via switchable normalization. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019 (2019). https://openreview.net/forum?id=ryggIs0cYQ
Mao, X., Li, Q., Xie, H., Lau, R.Y.K., Wang, Z., Smolley, S.P.: Least squares generative adversarial networks. In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, 22–29 October 2017, pp. 2813–2821 (2017). https://doi.org/10.1109/ICCV.2017.304
Nestmeyer, T., Lalonde, J.F., Matthews, I., Lehrmann, A.M.: Learning physics-guided face relighting under directional light. \({\rm arXiv}\): Computer Vision and Pattern Recognition (2020)
Philip, J., Gharbi, M., Zhou, T., Efros, A.A., Drettakis, G.: Multi-view relighting using a geometry-aware network. ACM Trans. Graph. (TOG) 38, 1–14 (2019)
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Sun, T., et al.: Single image portrait relighting. ACM Trans. Graph. (TOG) 38, 1–12 (2019)
Wang, P., et al.: Understanding convolution for semantic segmentation. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1451–1460 (2018)
Xie, S., Girshick, R.B., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5987–5995 (2017)
Xu, B., Wang, N., Chen, T., Li, M.: Empirical evaluation of rectified activations in convolutional network. CoRR abs/1505.00853 (2015). http://arxiv.org/abs/1505.00853
Zhou, H., Hadap, S., Sunkavalli, K., Jacobs, D.W.: Deep single-image portrait relighting. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7193–7201 (2019)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251 (2017)
Acknowledgements
This work was supported by the Advance Research Program (31511130301); National Key Research and Development Program (2017YFF0209806), and National Natural Science Foundation of China (No. 61906193; No. 61906195; No. 61702510).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Dong, L. et al. (2020). An Ensemble Neural Network for Scene Relighting with Light Classification. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science(), vol 12537. Springer, Cham. https://doi.org/10.1007/978-3-030-67070-2_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-67070-2_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-67069-6
Online ISBN: 978-3-030-67070-2
eBook Packages: Computer ScienceComputer Science (R0)