Skip to main content

Advertisement

SpringerLink
Log in

Sun position estimation and tracking for virtual object placement in time-lapse videos

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Realistic illumination of virtual objects placed in real videos is important in terms of achieving visual coherence. We propose a novel approach for illumination estimation on time-lapse videos and seamlessly insert virtual objects in these videos in a visually consistent way. The proposed approach works for both outdoor and indoor environments where the main light source is the Sun. We first modify an existing illumination estimation method that aims to obtain sparse radiance map of the environment in order to estimate the initial Sun position. We then track the hard ground shadows on the time-lapse video by using an energy-based pixel-wise method. The proposed method aims to track the shadows by utilizing the energy values of the pixels that forms them. We tested the method on various time-lapse videos recorded in outdoor and indoor environments and obtained successful results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Panagopoulos, A., Wang, C., Samaras, D., Paragios, N.: Illumination estimation and cast shadow detection through a higher-order graphical model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11), pp. 673–680 (2011)

  2. Lalonde, J.-F., Efros, A.A., Narasimhan, S.G.: Estimating the natural illumination conditions from a single outdoor image. Int. J. Comput. Vis. 98(2), 123–145 (2012)

    Article  MathSciNet  Google Scholar 

  3. Guo, R., Dai, Q., Hoiem, D.: Paired regions for shadow detection and removal. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2956–2967 (2013)

    Article  Google Scholar 

  4. Liu, Y., Gevers, T., Li, X.: Estimation of sunlight direction using 3D object models. IEEE Trans. Image Proc. 24(3), 932–942 (2014)

    MathSciNet  Google Scholar 

  5. Lalonde, J.-F., Matthews, I.: Lighting estimation in outdoor image collections. In: Proceedings of the International Conference on 3D Vision, vol. 1, Tokyo, Japan, pp. 131–138 (2014)

  6. Liu, Y., Granier, X.: Online tracking of outdoor lighting variations for augmented reality with moving cameras. IEEE Trans. Vis. Comput. Graph. 18(4), 573–580 (2012)

    Article  Google Scholar 

  7. Andersen, M.S., Jensen, T., Madsen, C.B.: Estimation of dynamic light changes in outdoor scenes without the use of calibration objects. In: Proceedings of the International Conference on Pattern Recognition, vol. 4, 91–94 (2006)

  8. Xing, G., Liu, Y., Qin, X., Peng, Q.: On-line illumination estimation of outdoor scenes based on area selection for augmented reality. In: Proceedings of the 12th International Conference on Computer-Aided Design and Computer Graphics (CADGRAPHICS’11), pp. 439–442 (2011)

  9. Ikeda, T., Oyamada, Y., Sugimoto, M., Saito, H.: Illumination estimation from shadow and incomplete object shape captured by an RGB-D camera. In: Proceedings of the 21st International Conference on Patten Recognition Series (ICPR’12), Tsukuba, Japan, pp. 165–169 (2012)

  10. Gruber, L., Langlotz, T., Sen, P., Hoherer, T., Schmalstieg, D.: Efficient and robust radiance transfer for probeless photorealistic augmented reality. In: Proceedings of the IEEE Virtual Reality, pp. 15–20 (2014)

  11. Lensing, P., Broll, W.: Instant indirect illumination for dynamic mixed reality scenes. In: Proceedings of the 11th IEEE International Symposium on Mixed and Augmented Reality (ISMAR’12), pp. 109–118 (2012)

  12. Neverova, N., Muselet, D., Trémeau, A.: Lighting estimation in indoor environments from low-quality images. In: Proceedings of the 12th International Conference on Computer Vision (ICCV’12), vol. 2, pp. 380–389 (2012)

  13. Yoo, J., Lee, K.: Light source estimation for realistic shadow using segmented HDR images. In: Hong, D., Jeon, S. (eds.) Proceedings of the International Symposium on Ubiquitous Virtual Reality (ISUVR’07), vol. 260 (2007)

  14. Lopez-Moreno, J., Garces, E., Hadap, S., Reinhard, E., Gutierrez, D.: Multiple light source estimation in a single image. Comput. Graph. Forum 32(8), 170–182 (2013)

    Article  Google Scholar 

  15. Zhao, W., Zheng, Y., Wang, L., Peng, S.: Lighting estimation of a convex Lambertian object using weighted spherical harmonic frames. Signal Image Video Proc. 9(1), 57–75 (2015)

    Article  Google Scholar 

  16. Sunkavalli, K., Matusik, W., Pfister, H., Rusinkiewicz, S.: Factored time-lapse video. In: ACM Transactions on Graphics. Proceedings of the SIGGRAPH’07, vol. 26, no. 3, Article No. 101 (2007)

  17. Zhang, R., Zhong, F., Lin, L., Xing, G., Peng, Q., Qin, X.: Basis image decomposition of outdoor time-lapse videos. Vis. Comput. 29(11), 1197–1210 (2013)

    Article  Google Scholar 

  18. Lalonde, J.-F., Efros, A.A., Narasimhan, S.G.: Webcam clip art: Appearance and illuminant transfer from time-lapse sequences. In: ACM Transactions on Graphics. Proceedings of the SIGGRAPH Asia’09, vol. 28, no. 5, Article No. 131 (2009)

  19. Chen, X., Wang, K., Jin, X.: Single image based illumination estimation for lighting virtual object in real scene. In: Proceedings of the 12th International Conference on Computer-Aided Design and Computer Graphics, pp. 450–455 (2011)

  20. Saxena, A., Sun, M., Ng, A.: Make3D: learning 3D scene structure from a single still image. IEEE Trans. Pattern Anal. Mach. Intell. 31(5), 824–840 (2009)

    Article  Google Scholar 

  21. Garces, E., Munoz, A., Lopez-Moreno, J., Gutierrez, D.: Intrinsic images by clustering. In: Computer Graphics Forum. Proceedings of the Eurographics Symposium on Rendering, vol. 31, no. 4, pp. 1415–1424 (2012)

  22. Phong, B.T.: Illumination for computer generated pictures. Commun. ACM 18(6), 311–317 (1975)

    Article  Google Scholar 

  23. Moré, J.J.: The Levenberg–Marquardt algorithm: implementation and theory. In: Watson, G.A. (ed.) Numerical Analysis, pp. 105–116. Springer, Berlin (1977)

    Google Scholar 

  24. Lalonde, J.-F., Efros, A., Narasimhan, S.: Detecting ground shadows in outdoor consumer photographs. In: Proceedings of the European Conference on Computer Vision (ECCV’10), LNCS, vol. 6312, pp. 322–335 (2010)

  25. Avidan, S., Shamir, A.: Seam carving for content-aware image resizing. In: Proceedings of the 34th International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’07). New York, NY, USA: ACM (2007)

  26. Unity Technologies: Unity Game Engine. http://unity3d.com/. Accessed 2016-11-12

  27. MathWorks: MATLAB. http://www.mathworks.com/products/matlab/. Accessed 2016-11-12

  28. Gazda, T.: Sun’s shadow time lapse. https://www.youtube.com/watch?v=3B7KLstUZbI. Accessed 2016-11-12

  29. Davies, M.: Shadows timelapse. https://www.youtube.com/watch?v=Lvhjbrr5GI8. Accessed 2015-11-12

  30. Bates, G.: Fast food shadows timelapse. https://www.youtube.com/watch?v=mdhS6pds8VY. Accessed 2015-11-12

  31. Zheng, Y., Chen, X., Cheng, M.-M., Zhou, K., Hu, S.-M., Mitra, N.J.: Interactive images: Cuboid proxies for smart image manipulation. In: ACM Transactions on Graphics. Proceedings of the SIGGRAPH’12, vol. 31, no. 4, Article No. 99 (2012)

Download references

Acknowledgements

This research is supported by The Scientific and Technological Research Council of Turkey (TÜBİTAK) under Grant No. 112E110. The first author is supported by TÜBİTAK under BİDEB 2210 Graduate Scholarship. We gratefully acknowledge Gordon Bates, Matthew Davies and Tom Gazda for permitting us to use their videos.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Bilkent University, 06800, Bilkent, Ankara, Turkey

    Hasan Balcı & Uğur Güdükbay

Authors
  1. Hasan Balcı
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Uğur Güdükbay
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Uğur Güdükbay.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 889 KB)

Supplementary material 2 (mp4 64775 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Balcı, H., Güdükbay, U. Sun position estimation and tracking for virtual object placement in time-lapse videos. SIViP 11, 817–824 (2017). https://doi.org/10.1007/s11760-016-1027-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-016-1027-x

Keywords

Search

Navigation