skip to main content
research-article

Synthesis of Complex Image Appearance from Limited Exemplars

Published: 02 March 2015 Publication History

Abstract

Editing materials in photos opens up numerous opportunities like turning an unappealing dirt ground into luscious grass and creating a comfortable wool sweater in place of a cheap t-shirt. However, such edits are challenging. Approaches such as 3D rendering and BTF rendering can represent virtually everything, but they are also data intensive and computationally expensive, which makes user interaction difficult. Leaner methods such as texture synthesis are more easily controllable by artists, but also more limited in the range of materials that they handle, for example, grass and wool are typically problematic because of their non-Lambertian reflectance and numerous self-occlusions. We propose a new approach for editing of complex materials in photographs. We extend the texture-by-numbers approach with ideas from texture interpolation. The inputs to our method are coarse user annotation maps that specify the desired output, such as the local scale of the material and the illumination direction. Our algorithm then synthesizes the output from a discrete set of annotated exemplars. A key component of our method is that it can cope with missing data, interpolating information from the available exemplars when needed. This enables production of satisfying results involving materials with complex appearance variations such as foliage, carpet, and fabric from only one or a couple of exemplar photographs.

Supplementary Material

JPG File (a22.jpg)
MP4 File (a22.mp4)

References

[1]
C. Barnes, E. Shechtman, A. Finkelstein, and D. Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3.
[2]
C. Barnes, E. Shechtman, D. Goldman, and A. Finkelstein. 2010. The generalized patchmatch correspondence algorithm. In Proceedings of the European Conference on Computer Vision (ECCV'10). 29--43.
[3]
J. T. Barron and J. Malik. 2012. Color constancy, intrinsic images, and shape estimation. In Proceedings of the European Conference on Computer Vision (ECCV'12). A. W. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid, Eds., Lecture Notes in Computer Science, vol. 7575, Springer, 57--70.
[4]
P. Bhat, B. Curless, M. Cohen, and C. L. Zitnick. 2008. Fourier analysis of the 2D screened poisson equation for gradient domain problems. In Proceedings of the European Conference on Computer Vision (ECCV'08). 114--128.
[5]
N. Bonneel, M. Van De Panne, S. Lefebvre, and G. Drettakis. 2010. Proxy-guided texture synthesis for rendering natural scenes. In Proceedings of the International Workshop on Vision, Modeling and Visualization (VMV'10). 87--95.
[6]
D. Comaniciu and P. Meer. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Trans. Patt. Anal. Mach. Intell. 24, 5, 603--619.
[7]
K. J. Dana, B. Van Ginneken, S. K. Nayar, and J. J. Koenderink. 1999. Reflectance and texture of real-world surfaces. ACM Trans. Graph. 18, 1, 1--34.
[8]
S. Darabi, E. Shechtman, C. Barnes, D. B. Goldman, and P. Sen. 2012. Image melding: Combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 31, 4.
[9]
A. A. Efros and T. K. Leung. 1999. Texture synthesis by nonparametric sampling. In Proceeding of the International Conference on Computer Vision (ICCV'99). Vol. 2. 1033--1038.
[10]
C. Eisenacher, S. Lefebvre, and M. Stamminger. 2008. Texture synthesis from photographs. Comput. Graph. Forum 27, 2, 419--428.
[11]
H. Fang and J. C. Hart. 2004. Textureshop: Texture synthesis as a photograph editing tool. ACM Trans. Graph. 23, 3, 354--359.
[12]
C. Han, E. Risser, R. Ramamoorthi, and E. Grinspun. 2008. Multiscale texture synthesis. ACM Trans. Graph. 27, 3.
[13]
A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. 2001. Image analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'01). 327--340.
[14]
W. Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.
[15]
M. K. Johnson, K. Dale, S. Avidan, H. Pfister, W. T. Freeman, and W. Matusik. 2011. CG2Real: Improving the realism of computer generated images using a large collection of photographs. IEEE Trans. Visual. Comput. Graph. 17, 9, 1273--1285.
[16]
J. Kautz, S. Boulos, and F. Durand. 2007. Interactive editing and modeling of bidirectional texture functions. ACM Trans. Graph. 26, 3.
[17]
V. Kwatra, I. A. Essa, A. F. Bobick, and N. Kwatra. 2005. Texture optimization for example-based synthesis. ACM Trans. Graph. 24, 3, 795--802.
[18]
S. Lefebvre and H. Hoppe. 2005. Parallel controllable texture synthesis. ACM Trans. Graph. 24, 3, 777--786.
[19]
S. Lefebvre and H. Hoppe. 2006. Appearance-space texture synthesis. ACM Trans. Graph. 25, 3, 541--548.
[20]
D. Lepage and J. Lawrence. 2011. Material matting. ACM Trans. Graph. 30, 6, 144.
[21]
X. Liu, Y. Yu, and H.-Y. Shum. 2001. Synthesizing bidirectional texture functions for real-world surfaces. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'01). ACM Press, New York, 97--106.
[22]
J. Lu, A. S. Georghiades, A. Glaser, H. Wu, L.-Y. Wei, B. Guo, J. Dorsey, and H. E. Rushmeier. 2007. Context-aware textures. ACM Trans. Graph. 26, 1.
[23]
W. Matusik, M. Zwicker, and F. Durand. 2005. Texture design using a simplicial complex of morphable textures. ACM Trans. Graph. 24, 3, 787--794.
[24]
D. M. Mount and S. Arya. 1998. ANN: Library for approximate nearest neighbour searching. http://www.cs.umd.edu/-mount/ANN.
[25]
A. Ngan and F. Durand. 2006. Statistical acquisition of texture appearance. In Proceedings of the 17th Eurographics Conference on Rendering Techniques (EGSR'06). 31--40.
[26]
A. Orzan, A. Bousseau, H. Winnemoller, P. Barla, J. Thollot, and D. Salesin. 2008. Diffusion curves: A vector representation for smooth-shaded images. ACM Trans. Graph. 27, 3.
[27]
H. Park, H. Byun, and C.-H. Kim. 2013. Multi-exemplar inhomogeneous texture synthesis. Comput. Graph. 37, 1--2, 54--64.
[28]
E. Risser, C. Han, R. Dahyot, and E. Grinspun. 2010. Synthesizing structured image hybrids. ACM Trans. Graph. 29, 4.
[29]
A. Rosenberger, D. Cohen-Or, and D. Lischinski. 2009. Layered shape synthesis: Automatic generation of control maps for non-stationary textures. ACM Trans. Graph. 28, 5.
[30]
R. Ruiters, C. Schwartz, and R. Klein. 2013. Example-based interpolation and synthesis of bidirectional texture functions. Comput. Graph. Forum 32, 2.
[31]
J. Wang, X. Tong, S. Lin, M. Pan, C. Wang, H. Bao, B. Guo, and H.-Y. Shum. 2006. Appearance manifolds for modeling time-variant appearance of materials. ACM Trans. Graph. 25, 3, 754--761.
[32]
L.-Y. Wei and M. Levoy. 2000. Fast texture synthesis using tree-structured vector quantization. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'00). 479--488.
[33]
J. Zhang, K. Zhou, L. Velho, B. Guo, and H.-Y. Shum. 2003. Synthesis of progressively-variant textures on arbitrary surfaces. ACM Trans. Graph. 22, 3, 295--302.

Cited By

View all
  • (2024)Analogist: Out-of-the-box Visual In-Context Learning with Image Diffusion ModelACM Transactions on Graphics10.1145/365813643:4(1-15)Online publication date: 19-Jul-2024
  • (2024)Material Palette: Extraction of Materials from a Single Image2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00419(4379-4388)Online publication date: 16-Jun-2024
  • (2023)Diffusion Image AnalogiesACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591558(1-10)Online publication date: 23-Jul-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 34, Issue 2
February 2015
136 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/2742222
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 March 2015
Accepted: 01 October 2014
Revised: 01 August 2014
Received: 01 February 2014
Published in TOG Volume 34, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Texture synthesis
  2. image-based editing
  3. material editing
  4. missing data
  5. natural material synthesis
  6. texture interpolation

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

  • ERC Starting Grant iModel (StG-2012-306877)

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)20
  • Downloads (Last 6 weeks)4
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Analogist: Out-of-the-box Visual In-Context Learning with Image Diffusion ModelACM Transactions on Graphics10.1145/365813643:4(1-15)Online publication date: 19-Jul-2024
  • (2024)Material Palette: Extraction of Materials from a Single Image2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00419(4379-4388)Online publication date: 16-Jun-2024
  • (2023)Diffusion Image AnalogiesACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591558(1-10)Online publication date: 23-Jul-2023
  • (2023)Neural Photometry-Guided Visual Attribute TransferIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.313308129:3(1818-1830)Online publication date: 1-Mar-2023
  • (2023)NeuBTF: Neural fields for BTF encoding and transferComputers & Graphics10.1016/j.cag.2023.06.018114(239-246)Online publication date: Aug-2023
  • (2023)Color TransferImage and Video Color Editing10.1007/978-3-031-26030-8_2(9-20)Online publication date: 21-Mar-2023
  • (2022)Gradient Terrain AuthoringComputer Graphics Forum10.1111/cgf.1446041:2(85-95)Online publication date: 24-May-2022
  • (2022)Inpainting at Modern Camera Resolution by Guided PatchMatch with Auto-curationComputer Vision – ECCV 202210.1007/978-3-031-19790-1_4(51-67)Online publication date: 23-Oct-2022
  • (2020)Lagrangian neural style transfer for fluidsACM Transactions on Graphics10.1145/3386569.339247339:4(52:1-52:10)Online publication date: 12-Aug-2020
  • (2020)Guided Fine‐Tuning for Large‐Scale Material TransferComputer Graphics Forum10.1111/cgf.1405639:4(91-105)Online publication date: 20-Jul-2020
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media