skip to main content
research-article

Decomposing time-lapse paintings into layers

Published: 27 July 2015 Publication History

Abstract

The creation of a painting, in the physical world or digitally, is a process that occurs over time. Later strokes cover earlier strokes, and strokes painted at a similar time are likely to be part of the same object. In the final painting, this temporal history is lost, and a static arrangement of color is all that remains. The rich literature for interacting with image editing history cannot be used. To enable these interactions, we present a set of techniques to decompose a time lapse video of a painting (defined generally to include pencils, markers, etc.) into a sequence of translucent "stroke" images. We present translucency-maximizing solutions for recovering physical (Kubelka and Munk layering) or digital (Porter and Duff "over" blending operation) paint parameters from before/after image pairs. We also present a pipeline for processing real-world videos of paintings capable of handling long-term occlusions, such as the painter's hand and its shadow, color shifts, and noise.

Supplementary Material

ZIP File (a61-tan.zip)
Supplemental files
MP4 File (a61.mp4)

References

[1]
Amati, C., and Brostow, G. J. 2010. Modeling 2.5D plants from ink paintings. In Proceedings of Eurographics Symposium on Sketch-Based Interfaces and Modeling Symposium, 41--48.
[2]
Barbarić -Mikočević, V. D.-M. Ž., and Itrić, K. 2011. Kubelka-Munk theory in describing optical properties of paper (i). Technical Gazette 18, 1, 117--124.
[3]
Baxter, W. V., Wendt, J., and Lin, M. C. 2004. IMPaSTo: A realistic, interactive model for paint. In Proceedings of the International Symposium on Non-Photorealistic Animation and Rendering, 45--56.
[4]
Berthouzoz, F., Li, W., Dontcheva, M., and Agrawala, M. 2011. A framework for content-adaptive photo manipulation macros: Application to face, landscape, and global manipulations. ACM Transactions on Graphics 30, 5, 120.
[5]
Bonanni, L., Xiao, X., Hockenberry, M., Subramani, P., Ishii, H., Seracini, M., and Schulze, J. 2009. Wetpaint: Scraping through multi-layered images. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 571--574.
[6]
Budsberg, J. B. 2007. Pigmented Colorants: Dependency on Media and Time. Master's thesis, Cornell Univrsity, Ithaca, New York, USA.
[7]
Chen, H.-T., Wei, L.-Y., and Chang, C.-F. 2011. Nonlinear revision control for images. ACM Transactions on Graphics 30, 4, 105.
[8]
Chen, T., Wei, L.-Y., Hartmann, B., and Agrawala, M. 2012. Data-driven history list for image editing. Tech. Rep. TR-2012-07, The University of Hong Kong.
[9]
Chen, H.-T., Grossman, T., Wei, L.-Y., Schmidt, R. M., Hartmann, B., Fitzmaurice, G., and Agrawala, M. 2014. History assisted view authoring for 3D models. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2027--2036.
[10]
Cheung, S.-c. S., and Kamath, C. 2004. Robust techniques for background subtraction in urban traffic video. In Proceedings of SPIE, vol. 5308, 881--892.
[11]
Curtis, C. J., Anderson, S. E., Seims, J. E., Fleischer, K. W., and Salesin, D. H. 1997. Computer-generated watercolor. In ACM SIGGRAPH Conference Proceedings, 421--430.
[12]
Dannen, C., 2012. The magical tech behind Paper for iPad's color-mixing perfection, Nov. Accessed: January 10th, 2015.
[13]
Denning, J. D., and Pellacini, F. 2013. MeshGit: Diffing and merging meshes for polygonal modeling. ACM Transactions on Graphics 32, 4, 35.
[14]
Farid, H., and Adelson, E. H. 1999. Separating reflections from images by use of independent component analysis. Journal of the Optical Society of America A 16, 9, 2136--2145.
[15]
Fu, H., Zhou, S., Liu, L., and Mitra, N. J. 2011. Animated construction of line drawings. ACM Transactions on Graphics 30, 6, 133.
[16]
Godbehere, A., Matsukawa, A., and Goldberg, K. 2012. Visual tracking of human visitors under variable-lighting conditions for a responsive audio art installation. In Proceedings of American Control Conference, 4305--4312.
[17]
Grabler, F., Agrawala, M., Li, W., Dontcheva, M., and Igarashi, T. 2009. Generating photo manipulation tutorials by demonstration. ACM Transactions on Graphics 28, 3, 66.
[18]
Grossman, T., Matejka, J., and Fitzmaurice, G. 2010. Chronicle: Capture, exploration, and playback of document workflow histories. In Proceedings of ACM Symposium on User Interface Software and Technology, 143--152.
[19]
Haase, C. S., and Meyer, G. W. 1992. Modeling pigmented materials for realistic image synthesis. ACM Transactions on Graphics 11, 4, 305--335.
[20]
Hu, S.-M., Xu, K., Ma, L.-Q., Liu, B., Jiang, B.-Y., and Wang, J. 2013. Inverse image editing: Recovering a semantic editing history from a before-and-after image pair. ACM Transactions on Graphics 32, 6, 194.
[21]
Hubbe, M. A., Pawlak, J. J., and Koukoulas, A. A. 2008. Paper's appearance: A review. BioResources 3, 2, 627--665.
[22]
Karsch, K., Golparvar-Fard, M., and Forsyth, D. 2014. Constructaide: Analyzing and visualizing construction sites through photographs and building models. ACM Transactions on Graphics 33, 6, 176.
[23]
Konieczny, J., and Meyer, G. 2009. Airbrush simulation for artwork and computer modeling. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering, 61--69.
[24]
Kubelka, P., and Munk, F. 1931. An article on optics of paint layers. Zeitschrift für Technische Physik 12, 593--601.
[25]
Kubelka, P. 1948. New contributions to the optics of intensely light-scattering materials. Part I. Journal of the Optical Society of America 38, 5, 448--448.
[26]
Kubelka, P. 1954. New contributions to the optics of intensely light-scattering materials. Part II: Nonhomogeneous layers. Journal of the Optical Society of America 44, 4, 330--334.
[27]
Lu, J., DiVerdi, S., Chen, W. A., Barnes, C., and Finkelstein, A. 2014. RealPigment: Paint compositing by example. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering, 21--30.
[28]
Matzen, K., and Snavely, N. 2014. Scene chronology. In Proceedings of European Conference on Computer Vision. 615--630.
[29]
McCann, J., and Pollard, N. 2009. Local layering. ACM Transactions on Graphics 28, 3, 84.
[30]
McCann, J., and Pollard, N. 2012. Soft stacking. Computer Graphics Forum 31, 2, 469--478.
[31]
Nancel, M., and Cockburn, A. 2014. Causality: A conceptual model of interaction history. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1777--1786.
[32]
Noris, G., Sýkora, D., Shamir, A., Coros, S., Whited, B., Simmons, M., Hornung, A., Gross, M., and Sumner, R. 2012. Smart scribbles for sketch segmentation. Computer Graphics Forum 31, 8, 2516--2527.
[33]
Novick, L. R., and Tversky, B. 1987. Cognitive constraints on ordering operations: The case of geometric analogies. Journal of Experimental Psychology: General 116, 1, 50.
[34]
Porter, T., and Duff, T. 1984. Compositing digital images. ACM SIGGRAPH Computer Graphics 18, 3, 253--259.
[35]
Richardt, C., Lopez-Moreno, J., Bousseau, A., Agrawala, M., and Drettakis, G. 2014. Vectorising bitmaps into semi-transparent gradient layers. Computer Graphics Forum (Proceedings of EGSR) 33, 4 (July), 11--19.
[36]
Rubinstein, M., Liu, C., Sand, P., Durand, F., and Freeman, W. T. 2011. Motion denoising with application to time-lapse photography. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 313--320.
[37]
Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In ACM SIGGRAPH Conference Proceedings, 259--268.
[38]
Su, S. L., Paris, S., and Durand, F. 2009. QuickSelect: History-based selection expansion. In Proceedings of Graphics Interface, 215--221.
[39]
Szeliski, R., Avidan, S., and Anandan, P. 2000. Layer extraction from multiple images containing reflections and transparency. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, 246--253.
[40]
Tange, O. 2011. GNU Parallel: The command-line power tool. ;login: The USENIX Magazine 36, 1 (Feb), 42--47.
[41]
Taylor, H. A., and Tversky, B. 1992. Descriptions and depictions of environments. Memory & Cognition 20, 5, 483--496.
[42]
Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proceedings of IEEE International Conference on Computer Vision, 839--846.
[43]
Tversky, B. 1999. What does drawing reveal about thinking? Visual and Spatial Reasoning in Design, 93--101.
[44]
van Sommers, P. 1984. Drawing and cognition: Descriptive and experimental studies of graphic production processes. Cambridge University Press.
[45]
VisTrails, I., 2009. Vistrails provenance explorer for maya. http://www.vistrails.com/maya.html.
[46]
Xing, J., Chen, H.-T., and Wei, L.-Y. 2014. Autocomplete painting repetitions. ACM Transactions on Graphics 33, 6, 172.
[47]
Xu, S., Xu, Y., Kang, S. B., Salesin, D. H., Pan, Y., and Shum, H.-Y. 2006. Animating chinese paintings through stroke-based decomposition. ACM Transactions on Graphics 25, 2, 239--267.
[48]
Xu, L., Lu, C., Xu, Y., and Jia, J. 2011. Image smoothing via L0 gradient minimization. ACM Transactions on Graphics 30, 6, 174.
[49]
Zivkovic, Z., and van der Heijden, F. 2006. Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recognition Letters 27, 7, 773--780.
[50]
Zongker, D. E., Werner, D. M., Curless, B., and Salesin, D. H. 1999. Environment matting and compositing. In ACM SIGGRAPH Conference Proceedings, 205--214.

Cited By

View all
  • (2024)STIVi: Turning Perspective Sketching Videos into Interactive TutorialsProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670969(1-13)Online publication date: 3-Jun-2024
  • (2024)Transparent Image Layer Diffusion using Latent TransparencyACM Transactions on Graphics10.1145/365815043:4(1-15)Online publication date: 19-Jul-2024
  • (2024)Artificial intelligence for geometry-based feature extraction, analysis and synthesis in artistic images: a surveyArtificial Intelligence Review10.1007/s10462-024-11051-358:2Online publication date: 21-Dec-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 34, Issue 4
August 2015
1307 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/2809654
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2015
Published in TOG Volume 34, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. channel
  2. depth
  3. images
  4. layers
  5. painting
  6. photoshop
  7. segmentation
  8. surfaces
  9. time
  10. video

Qualifiers

  • Research-article

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)39
  • Downloads (Last 6 weeks)5
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)STIVi: Turning Perspective Sketching Videos into Interactive TutorialsProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670969(1-13)Online publication date: 3-Jun-2024
  • (2024)Transparent Image Layer Diffusion using Latent TransparencyACM Transactions on Graphics10.1145/365815043:4(1-15)Online publication date: 19-Jul-2024
  • (2024)Artificial intelligence for geometry-based feature extraction, analysis and synthesis in artistic images: a surveyArtificial Intelligence Review10.1007/s10462-024-11051-358:2Online publication date: 21-Dec-2024
  • (2023)Image vectorization and editing via linear gradient layer decompositionACM Transactions on Graphics10.1145/359212842:4(1-13)Online publication date: 26-Jul-2023
  • (2023)RecolorNeRF: Layer Decomposed Radiance Fields for Efficient Color Editing of 3D ScenesProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611957(8004-8015)Online publication date: 26-Oct-2023
  • (2023)InfinitePaint: Painting in Virtual Reality with Passive Haptics Using Wet Brushes and a Physical Proxy CanvasProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580927(1-13)Online publication date: 19-Apr-2023
  • (2022)Stroke Transfer: Example-based Synthesis of Animatable Stroke StylesACM SIGGRAPH 2022 Conference Proceedings10.1145/3528233.3530703(1-10)Online publication date: 27-Jul-2022
  • (2021)Video recoloring via spatial-temporal geometric palettesACM Transactions on Graphics10.1145/3450626.345967540:4(1-16)Online publication date: 19-Jul-2021
  • (2021)Frequency-Aware SVD Decomposition and its Application to Color Magnification and Motion DenoisingIEEE Access10.1109/ACCESS.2021.31018239(108832-108845)Online publication date: 2021
  • (2020)Accurate Overlapping Method of Ultra-Long Interval Time-Lapse Images for World Heritage Site InvestigationJournal on Computing and Cultural Heritage 10.1145/337335713:2(1-18)Online publication date: 30-May-2020
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media