Abstract
When a virtual object is inserted into an outdoor image, the recovery of scene illumination has a critical effect on the mix of virtual objects and actual reality. There are two main parts of the object in the outdoor scene: the sun and the sky. In order to represent the illumination conditions of these two natural illumination, this paper uses the Lalonde-Matthew outdoor illumination model to perform the sky and sun in the image. Model use seven parameters represent the illumination of the scene. So the original illumination estimation problem is transformed into a prediction problem of seven illumination parameters. For this problem, this paper proposes a new two-branch network structure, one branch is used to estimate the sun orientation, and the other branch is used to estimate the remaining six parameters. This paper also introduces convolution block attention module (CBAM) based on this structure. The introduction of this module enables the network to select the most important information for the current task target from a large number of information when extracting image features, while suppressing other useless information.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Barron JT, Malik J (2014) Shape, illumination, and reflectance from shading. IEEE Trans Pattern Anal Mach Intell 37(8):1670–1687
Cheng D, Shi J, Chen Y, Deng X, Zhang X (2018) Learning scene illumination by pairwise photos from rear and front mobile cameras. Comput Graph Forum, Wiley Online Library 37:213–221
Debevec P (2008) Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: ACM SIGGRAPH 2008 classes, pp 1–10
Erra U, Capece N (2019) Engineering an advanced geo-location augmented reality framework for smart mobile devices. J Ambient Intell Hum Comput 10(1):255–265
Gardner MA, Sunkavalli K, Yumer E, Shen X, Gambaretto E, Gagné C, Lalonde JF (2017) Learning to predict indoor illumination from a single image. arXiv preprint arXiv:170400090
Green R (2003) Spherical harmonic lighting: the gritty details. In: Archives of the game developers conference, vol 56, p 4
Hold-Geoffroy Y, Sunkavalli K, Hadap S, Gambaretto E, Lalonde JF (2017) Deep outdoor illumination estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7312–7321
Hosek L, Wilkie A (2012) An analytic model for full spectral sky-dome radiance. ACM Trans Graph (TOG) 31(4):1–9
HošekHošek L, Wilkie A (2013) Adding a solar-radiance function to the hošek-wilkie skylight model. IEEE Comput Graph Appl 33(3):44–52
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141
Jaderberg M, Simonyan K, Zisserman A, et al. (2015) Spatial transformer networks. In: Advances in neural information processing systems, pp 2017–2025
Jin X, Sun X, Zhang X, Sun H, Xu R, Zhou X, Li X, Liu R (2019) Sun orientation estimation from a single image using short-cuts in dcnn. Opt Laser Technol 110:191–195
Kim T, Hong KS (2005) A practical single image based approach for estimating illumination distribution from shadows. In: Tenth IEEE international conference on computer vision (ICCV’05) Volume 1, IEEE, vol 1, pp 266–271
Lalonde JF, Matthews I (2014) Lighting estimation in outdoor image collections. In: 2014 2nd International conference on 3D vision, IEEE, vol 1, pp 131–138
Lalonde JF, Efros AA, Narasimhan SG (2012) Estimating the natural illumination conditions from a single outdoor image. Int J Comput Vis 98(2):123–145
Lele A (2013) Virtual reality and its military utility. J Ambient Intell Hum Comput 4(1):17–26
Li Y, Lu H, Serikawa S (2015) Underwater image devignetting and colour correction. International conference on image and graphics, pp 510–521
Li Y, Lu H, Li K, Kim H, Serikawa S (2018) Non-uniform de-scattering and de-blurring of underwater images. Mob Netw Appl 23(2):352–362
Li P, Chen B, Ouyang W, Wang D, Yang X, Lu H (2019) Gradnet: Gradient-guided network for visual object tracking. In: Proceedings of the IEEE international conference on computer vision, pp 6162–6171
Lombardi S, Nishino K (2015) Reflectance and illumination recovery in the wild. IEEE Trans Pattern Anal Mach Intell 38(1):129–141
Lu H, Li Y, Nakashima S, Serikawa S (2016) Turbidity underwater image restoration using spectral properties and light compensation. IEICE Trans Inf Syst 99(1):219–227
Ma WC, Wang S, Brubaker MA, Fidler S, Urtasun R (2017) Find your way by observing the sun and other semantic cues. In: 2017 IEEE international conference on robotics and automation (ICRA), IEEE, pp 6292–6299
Perez R, Seals R, Michalsky J (1993) All-weather model for sky luminance distribution-preliminary configuration and validation. Solar Energy 50(3):235–245
Preetham AJ, Shirley P, Smits B (1999) A practical analytic model for daylight. In: Proceedings of the 26th annual conference on computer graphics and interactive techniques, pp 91–100
Stumpfel J, Jones A, Wenger A, Tchou C, Hawkins T, Debevec P (2006) Direct hdr capture of the sun and sky. In: ACM SIGGRAPH 2006 Courses, pp 5–es
Wang M, Callaghan V, Bernhardt J, White K, Peña-Rios A (2018a) Augmented reality in education and training: pedagogical approaches and illustrative case studies. J Ambient Intell Hum Comput 9(5):1391–1402
Wang X, Girshick R, Gupta A, He K (2018) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794–7803
Weber H, Prévost D, Lalonde JF (2018) Learning to estimate indoor lighting from 3d objects. In: 2018 International conference on 3D vision (3DV), IEEE, pp 199–207
Woo S, Park J, Lee JY, So Kweon I (2018) Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19
Xiao J, Ehinger KA, Oliva A, Torralba A (2012) Recognizing scene viewpoint using panoramic place representation. In: 2012 IEEE conference on computer vision and pattern recognition, IEEE, pp 2695–2702
Yan B, Zhao H, Wang D, Lu H, Yang X (2019) ‘Skimming-perusal’ tracking: a framework for real-time and robust long-term tracking. In: Proceedings of the IEEE international conference on computer vision, pp 2385–2393
Zhang H, Goodfellow I, Metaxas D, Odena A (2018) Self-attention generative adversarial networks. arXiv preprint arXiv:180508318
Zhang J, Sunkavalli K, Hold-Geoffroy Y, Hadap S, Eisenman J, Lalonde JF (2019) All-weather deep outdoor lighting estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 10158–10166
Zhao Q (2009) A survey on virtual reality. Sci China Ser F Inf Sci 52(3):348–400
Zhou Z, Zhou Y, Xiao J (2015) Survey on augmented virtual environment and augmented reality. SCIENTIA SINICA Inf 45(2):157–180
Acknowledgements
This work is partially supported by the National Natural Science Foundation of China (Grant Numbers 61701008, 61772047), the Beijing Natural Science Foundation (Grant Number 19L2040), the Open Project Program of State Key Laboratory of Cryptology (Grant Number MMKFKT201804), the Open Project Program of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (Grant Number VRLAB2019C03) and the Fundamental Research Funds for the Central Universities (Grant Number 328201907).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Jin, X., Deng, P., Li, X. et al. Sun-sky model estimation from outdoor images. J Ambient Intell Human Comput 13, 5151–5162 (2022). https://doi.org/10.1007/s12652-020-02367-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-020-02367-3