skip to main content
research-article

Webcam clip art: appearance and illuminant transfer from time-lapse sequences

Published: 01 December 2009 Publication History

Abstract

Webcams placed all over the world observe and record the visual appearance of a variety of outdoor scenes over long periods of time. The recorded time-lapse image sequences cover a wide range of illumination and weather conditions -- a vast untapped resource for creating visual realism. In this work, we propose to use a large repository of webcams as a "clip art" library from which users may transfer scene appearance (objects, scene backdrops, outdoor illumination) into their own time-lapse sequences or even single photographs. The goal is to combine the recent ideas from data-driven appearance transfer techniques with a general and theoretically-grounded physically-based illumination model. To accomplish this, the paper presents three main research contributions: 1) a new, high-quality outdoor webcam database that has been calibrated radiometrically and geometrically; 2) a novel approach for matching illuminations across different scenes based on the estimation of the properties of natural illuminants (sun, sky, weather and clouds), the camera geometry, and illumination-dependent scene features; 3) a new algorithm for generating physically plausible high dynamic range environment maps for each frame in a webcam sequence.

References

[1]
Arnold, D., Chalmers, A., Niccolucci, F., Stumpfel, J., Tchou, C., Yun, N., Hawkins, T., Jones, A., Emerson, B., and Debevec, P. 2003. Digital reunification of the parthenon and its sculptures. In 4th International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage.
[2]
Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., and Nayar, S. K. 2008. Face swapping: automatically replacing faces in photographs. ACM Transactions on Graphics (SIGGRAPH) 27, 3, 39.
[3]
Debevec, P., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach. In ACM SIGGRAPH.
[4]
Debevec, P., Wenger, A., Tchou, C., Gardner, A., Waese, J., and Hawkins, T. 2002. A lighting reproduction approach to live-action compositing. ACM Transactions on Graphics (SIGGRAPH).
[5]
Debevec, P. 1998. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In ACM SIGGRAPH.
[6]
Dror, R. O., Willsky, A. S., and Adelson, E. H. 2004. Statistical characterization of real-world illumination. Journal of Vision 4.
[7]
Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In ACM SIGGRAPH.
[8]
Goesele, M., Snavely, N., Curless, B., Hoppe, H., and Seitz, S. M. 2007. Multi-view stereo for community photo collections. In International Conference on Computer Vision.
[9]
Haber, T., Fuchs, C., Bekaert, P., Seidel, H.-P., Goesele, M., and Lensch, H. P. A. 2009. Relighting objects from image collections. In Conference on Computer Vision and Pattern Recognition.
[10]
Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Transactions on Graphics (SIGGRAPH) 26, 3, 4.
[11]
Hoiem, D., Efros, A. A., and Hebert, M. 2005. Geometric context from a single image. In International Conference on Computer Vision.
[12]
Horry, Y., Anjyo, K.-I., and Arai, K. 1997. Tour into the picture: using a spidery mesh interface to make animation from a single image. In ACM SIGGRAPH.
[13]
Jacobs, N., Roman, N., and Pless, R. 2007. Consistent temporal variations in many outdoor scenes. In Conference on Computer Vision and Pattern Recognition.
[14]
Jacobs, N., Satkin, S., Roman, N., Speyer, R., and Pless, R. 2007. Geolocating static cameras. In International Conference on Computer Vision.
[15]
Jacobs, N., Roman, N., and Pless, R. 2008. Toward fully automatic geo-location and geo-orientation of static outdoor cameras. In Workshop on applications of computer vision.
[16]
Kasten, F., and Young, A. T. 1989. Revised optical air mass tables and approximation formula. Applied optics 28.
[17]
Khan, E. A., Reinhard, E., Fleming, R., and Büelthoff, H. 2006. Image-based material editing. ACM Transactions on Graphics (SIGGRAPH).
[18]
Kim, S. J., Frahm, J.-M., and Polleyfeys, M. 2008. Radiometric calibration with illumination change for outdoor scene analysis. In Conference on Computer Vision and Pattern Recognition.
[19]
Koppal, S. J., and Narasimhan, S. G. 2006. Clustering appearance for scene analysis. In International Conference on Computer Vision.
[20]
Kuthirummal, S., Agarwala, A., Glodman, D. B., and Nayar, S. K. 2008. Priors for large photo collections and what they reveal about cameras. In European Conference on Computer Vision.
[21]
Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Transactions on Graphics (SIGGRAPH) 26, 3, 3.
[22]
Lalonde, J.-F., Narasimhan, S. G., and Efros, A. A. 2009. What do the sun and the sky tell us about the camera? International Journal on Computer Vision (to appear).
[23]
Lin, S., Gu, J., Yamazaki, S., and Shum, H.-Y. 2004. Radiometric calibration from a single image. In Conference on Computer Vision and Pattern Recognition.
[24]
Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 2.
[25]
Opperman, I., Ferndriger, S., and Eugster, J., 2009. Touristic webcams worldwide. Accessed: 09/03/09, http://www.webcams.travel.
[26]
Perez, R., Seals, R., and Michalsky, J. 1993. All-weather model for sky luminance distribution -- preliminary configuration and validation. Solar Energy 50, 3 (March), 235--245.
[27]
Preetham, A. J., Shirley, P., and Smits, B. 1999. A practical analytic model for daylight. In ACM SIGGRAPH.
[28]
Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Computer Graphics and Applications, special issue on Applied Perception 21, 5 (September--October), 34--41.
[29]
Schöld, A., Szeliski, R., Salesin, D., and Essa, I. 2000. Video textures. In ACM SIGGRAPH.
[30]
Snavely, N., Garg, R., Seitz, S. M., and Szeliski, R. 2008. Finding paths through the world's photos. ACM Transactions on Graphics (SIGGRAPH) 27, 3, 15.
[31]
Stumpfel, J., Jones, A., Wenger, A., Tchou, C., Hawkins, T., and Debevec, P. 2004. Direct HDR capture of the sun and sky. In AFRIGRAPH.
[32]
Sunkavalli, K., Matusik, W., Pfister, H., and Rusinkiewicz, S. 2007. Factored time-lapse video. ACM Transactions on Graphics (SIGGRAPH) 26, 3, 101.
[33]
Sunkavalli, K., Romeiro, F., Matusik, W., Zickler, T., and Pfister, H. 2008. What do color changes reveal about an outdoor scene? In Conference on Computer Vision and Pattern Recognition.
[34]
Tao, L., Yuan, L., and Sun, J. 2009. SkyFinder: attribute-based sky image search. ACM Transactions on Graphics (SIGGRAPH) 28, 3, 68.
[35]
Weiss, Y. 2001. Deriving intrinsic images from image sequences. In International Conference on Computer Vision.

Cited By

View all
  • (2022)Autoencoder and Partially Impossible Reconstruction LossesSensors10.3390/s2213486222:13(4862)Online publication date: 27-Jun-2022
  • (2022)WALT: Watch And Learn 2D amodal representation from Time-lapse imagery2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.00914(9346-9356)Online publication date: Jun-2022
  • (2022)What Does Learning About Time Tell About Outdoor Scenes?2022 IEEE Applied Imagery Pattern Recognition Workshop (AIPR)10.1109/AIPR57179.2022.10092235(1-6)Online publication date: 11-Oct-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 28, Issue 5
December 2009
646 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/1618452
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 2009
Published in TOG Volume 28, Issue 5

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. HDR
  2. computer vision
  3. image databases
  4. image-based lighting
  5. object insertion
  6. time-lapse video

Qualifiers

  • Research-article

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)24
  • Downloads (Last 6 weeks)3
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Autoencoder and Partially Impossible Reconstruction LossesSensors10.3390/s2213486222:13(4862)Online publication date: 27-Jun-2022
  • (2022)WALT: Watch And Learn 2D amodal representation from Time-lapse imagery2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.00914(9346-9356)Online publication date: Jun-2022
  • (2022)What Does Learning About Time Tell About Outdoor Scenes?2022 IEEE Applied Imagery Pattern Recognition Workshop (AIPR)10.1109/AIPR57179.2022.10092235(1-6)Online publication date: 11-Oct-2022
  • (2022)NeRF for Outdoor Scene RelightingComputer Vision – ECCV 202210.1007/978-3-031-19787-1_35(615-631)Online publication date: 23-Oct-2022
  • (2022)Animation Generation Technology Based on Deep Learning: Opportunities and ChallengesArtificial Intelligence and Security10.1007/978-3-031-06794-5_26(311-325)Online publication date: 15-Jul-2022
  • (2021)Illumination Normalization by Partially Impossible Encoder-Decoder Cost Function2021 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV48630.2021.00150(1458-1467)Online publication date: Jan-2021
  • (2021)Learning to Predict Repeatability of Interest Points2021 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA48506.2021.9561383(10294-10301)Online publication date: 30-May-2021
  • (2021)Frequency-Aware SVD Decomposition and its Application to Color Magnification and Motion DenoisingIEEE Access10.1109/ACCESS.2021.31018239(108832-108845)Online publication date: 2021
  • (2021)Outdoor illumination estimation via all convolutional neural networksComputers & Electrical Engineering10.1016/j.compeleceng.2021.10698790(106987)Online publication date: Mar-2021
  • (2020)Distributed Sky Imaging Radiometry and Tomography2020 IEEE International Conference on Computational Photography (ICCP)10.1109/ICCP48838.2020.9105241(1-12)Online publication date: Apr-2020
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media