skip to main content
research-article

Generating Digital Painting Lighting Effects via RGB-space Geometry

Published: 24 February 2020 Publication History

Abstract

We present an algorithm to generate digital painting lighting effects from a single image. Our algorithm is based on a key observation: Artists use many overlapping strokes to paint lighting effects, i.e., pixels with dense stroke history tend to gather more illumination strokes. Based on this observation, we design an algorithm to both estimate the density of strokes in a digital painting using color geometry and then generate novel lighting effects by mimicking artists’ coarse-to-fine workflow. Coarse lighting effects are first generated using a wave transform and then retouched according to the stroke density of the original illustrations into usable lighting effects.
Our algorithm is content-aware, with generated lighting effects naturally adapting to image structures, and can be used as an interactive tool to simplify current labor-intensive workflows for generating lighting effects for digital and matte paintings. In addition, our algorithm can also produce usable lighting effects for photographs or three-dimensional rendered images. We evaluate our approach with both an in-depth qualitative and a quantitative analysis that includes a perceptual user study. Results show that our proposed approach is not only able to produce favorable lighting effects with respect to existing approaches, but also that it is able to significantly reduce the needed interaction time.

References

[1]
Elad Aharoni-Mack, Yakov Shambik, and Dani Lischinski. 2017. Pigment-based recoloring of watercolor paintings. In Proceedings of the Symposium on Non-Photorealistic Animation and Rendering. Association for Computing Machinery, Los Angeles, California, 11.
[2]
Yagiz Aksoy, Tung Ozan Aydin, Aljosa Smolic, and Marc Pollefeys. 2017. Unmixing-based soft color segmentation for image manipulation. ACM Trans. Graph. 36, 2, Article 19 (March 2017), 19 pages.
[3]
Yağız Aksoy, Changil Kim, Petr Kellnhofer, Sylvain Paris, Mohamed Elgharib, Marc Pollefeys, and Wojciech Matusik. 2018. A dataset of flash and ambient illumination pairs from the crowd. In Computer Vision – ECCV 2018, Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer International Publishing, Cham, 644--660.
[4]
C. Bradford Barber, David P. Dobkin, and Hannu Huhdanpaa. 1996. The quickhull algorithm for convex hulls. ACM Trans. Math. Softw. 22, 4 (Dec. 1996), 469--483.
[5]
J. T. Barron, J. Malik, Jonathan T. Barron, and Jitendra Malik. 2012. Color constancy, intrinsic images, and shape estimation. In Computer Vision -- ECCV 2012, Andrew Fitzgibbon, Svetlana Lazebnik, Pietro Perona, Yoichi Sato, and Cordelia Schmid (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 57--70.
[6]
H. G. Barrow and J. M. Tenenbaum. 1978. Recovering Intrinsic Scene Characteristics from Images, Vol. 1. Academic Press, Chapter 1, 3--26.
[7]
Sean Bell, Kavita Bala, and Noah Snavely. 2014. Intrinsic images in the wild. ACM Trans Graph 33, 4, Article 159 (July 2014), 12 pages.
[8]
Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2013. OpenSurfaces: A richly annotated catalog of surface appearance. ACM Trans Graph 32, 4, Article 111 (July 2013), 17 pages.
[9]
Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material recognition in the wild with the materials in context database. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’15). 3479--3487.
[10]
Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein. 2015. Palette-based photo recoloring. ACM Trans. Graph. 34, 4, Article 139 (July 2015), 11 pages.
[11]
Jiansheng Chen, Guangda Su, Jinping He, and Shenglan Ben. 2010. Face image relighting using locally constrained global optimization. In Computer Vision--ECCV 2010, Kostas Daniilidis, Petros Maragos, and Nikos Paragios (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 44--57.
[12]
Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’00). ACM Press/Addison-Wesley Publishing Co., USA, 145--156.
[13]
G. Finlayson, H. Gong, and R. B. Fisher. 2019. Color homography: Theory and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 1 (Jan 2019), 20--33.
[14]
K. He, G. Gkioxari, P. Dollár, and R. Girshick. 2017. Mask R-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV’17). 2980--2988.
[15]
Matis Hudon, Mairéad Grogan, Rafael Pagés, and Aljoša Smolić. 2019. Deep normal estimation for automatic shading of hand-drawn characters. In Computer Vision -- ECCV 2018 Workshops, Laura Leal-Taixé and Stefan Roth (Eds.). Springer International Publishing, Cham, 246--262.
[16]
Yoshihiro Kanamori and Yuki Endo. 2018. Relighting humans: Occlusion-aware inverse rendering for full-body human images. ACM Transactions on Graphics 37, 6, Article 270 (Dec. 2018), 11 pages.
[17]
John R. Kender and Earl M. Smith. 1992. Shape from Darkness: Deriving Surface Information from Dynamic Shadows. Jones and Bartlett Publishers, Inc., USA, Chapter 3, 378--385.
[18]
Yuki Koyama and Masataka Goto. 2018. Decomposing images into layers with advanced color blending. Computer Graphics Forum 37, 7 (2018), 397--407.
[19]
Sharon Lin, Matthew Fisher, Angela Dai, and Pat Hanrahan. 2017. LayerBuilder: Layer decomposition for interactive image and video color editing. Arxiv 1701, 3754 (2017), 1701.03754.
[20]
Wojciech Matusik, Matthew Loper, and Hanspeter Pfister. 2004. Progressively-refined reflectance functions from natural illumination. In Proceedings of the Fifteenth Eurographics Conference on Rendering Techniques (EGSR04’04). Eurographics Association, Goslar, DEU, 299--308.
[21]
Pieter Peers and Philip Dutre. 2005. Inferring reflectance functions from wavelet noise. In Proceedings of the Sixteenth Eurographics Conference on Rendering Techniques (EGSR’05). Eurographics Association, Goslar, DEU, 173--182.
[22]
Pieter Peers, Dhruv K. Mahajan, Bruce Lamond, Abhijeet Ghosh, Wojciech Matusik, Ravi Ramamoorthi, and Paul Debevec. 2009. Compressive light transport sensing. ACM Trans. Graph. 28, 1, Article 3 (Feb. 2009), 18 pages.
[23]
Pieter Peers, Naoki Tamura, Wojciech Matusik, and Paul Debevec. 2007. Post-production facial performance relighting using reflectance transfer. ACM Trans. Graph. 26, 3 (July 2007), 52--es.
[24]
Julien Philip, Michael Gharbi, Tinghui Zhou, Alexei, Efros, and George Drettakis. 2019. Multi-view relighting using a geometry-aware network. ACM Trans. Graph. 38, 4, Article 78 (July 2019), 14 pages.
[25]
Carsten Rother, Martin Kiefel, Lumin Zhang, Bernhard Schölkopf, and Peter V. Gehler. 2011. Recovering intrinsic images with a global sparsity prior on reflectance. In Advances in Neural Information Processing Systems 24, J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 765--773.
[26]
Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, and David W. Jacobs. 2018. SfSNet: Learning shape, refectance and illuminance of faces in the wild. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6296--6305.
[27]
M. Serra, O. Penacchio, R. Benavente, and M. Vanrell. 2012. Names and shades of color for intrinsic image estimation. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’12). 278--285.
[28]
Jianbing Shen, Xiaoshan Yang, Yunde Jia, and Xuelong Li. 2011. Intrinsic images using optimization. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’11). 3481--3487.
[29]
YiChang Shih, Sylvain Paris, Fredo Durand, and William T. Freeman. 2013. Data-driven hallucination for different times of day from a single outdoor photo. ACM Trans. Graph. 32, 6, Article 200 (Nov. 2013), 11 pages.
[30]
Wanchao Su, Dong Du, Xin Yang, Shizhe Zhou, and Hongbo Fu. 2018. Interactive sketch-based normal map generation with deep neural networks. ACM Comput. Graph. Interact. Techn. 1, 1 (Jul 2018), 1–17.
[31]
Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi. 2019. Single image portrait relighting. ACM Trans. Graph. 38, 4, Article 79 (July 2019), 12 pages.
[32]
Daniel Sykora, Ladislav Kavan, Martin Cadik, Ondrej Jamriska, Alec Jacobson, Brian Whited, Maryann Simmons, and Olga Sorkine-Hornung. 2014. Ink-and-ray: Bas-relief meshes for adding global illumination effects to hand-drawn characters. ACM Trans. Graph. 33, 2, Article 16 (Apr. 2014), 1–15.
[33]
Jianchao Tan, Jose Echevarria, and Yotam Gingold. 2018. Efficient palette-based decomposition and recoloring of images via RGBXY-space geometry. ACM Trans. Graph. 37, 6, Article 262 (Dec. 2018), 10 pages.
[34]
Jianchao Tan, Jyh-Ming Lien, and Yotam Gingold. 2016. Decomposing images into layers via RGB-space geometry. ACM Trans. Graph. 36, 4, Article 61a (Nov. 2016), 14 pages.
[35]
Tai-Pang Wu, Jian Sun, Chi-Keung Tang, and Heung-Yeung Shum. 2008. Interactive normal reconstruction from a single image. ACM Trans. Graph. 27, 5, Article 119 (Dec. 2008), 9 pages.
[36]
Tai-Pang Wu, Chi-Keng Tang, Michael S. Brown, and Heung-Yeung Shum. 2007. ShapePalettes: Interactive normal transfer via sketching. ACM Trans. Graph. 26, 3 (July 2007), 44--es.
[37]
Qiuying Xu, Yotam Gingold, and Karan Singh. 2015. Inverse toon shading: Interactive normal field modeling with isophotes. In Proceedings of the Workshop on Sketch-Based Interfaces and Modeling (SBIM’15). Eurographics Association, Goslar, DEU, 15--25.
[38]
Ye Yu and William A. P. Smith. 2019. InverseRenderNet: Learning single image inverse rendering. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’19). 3150--3159.
[39]
Qing Zhang, Chunxia Xiao, Hanqiu Sun, and Feng Tang. 2017. Palette-based image recoloring using color decomposition optimization. IEEE Trans. Image Process. 26, 4 (April 2017), 1952--1964.

Cited By

View all
  • (2024)Collaboration and Transition: Distilling Item Transitions into Multi-Query Self-Attention for Sequential RecommendationProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635787(1003-1011)Online publication date: 4-Mar-2024
  • (2022)Visual Planarization in Oil Painting Techniques in Digital Information AgeInternational Transactions on Electrical Energy Systems10.1155/2022/99392092022(1-10)Online publication date: 22-Sep-2022
  • (2022)Fast Nonlinear Image Unblending2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV51458.2022.00325(3193-3201)Online publication date: Jan-2022
  • Show More Cited By

Index Terms

  1. Generating Digital Painting Lighting Effects via RGB-space Geometry

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Graphics
      ACM Transactions on Graphics  Volume 39, Issue 2
      April 2020
      136 pages
      ISSN:0730-0301
      EISSN:1557-7368
      DOI:10.1145/3381407
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 24 February 2020
      Accepted: 01 January 2020
      Revised: 01 December 2019
      Received: 01 September 2019
      Published in TOG Volume 39, Issue 2

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Relighting
      2. color
      3. convex hull

      Qualifiers

      • Research-article
      • Research
      • Refereed

      Funding Sources

      • JST PRESTO

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)126
      • Downloads (Last 6 weeks)4
      Reflects downloads up to 20 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Collaboration and Transition: Distilling Item Transitions into Multi-Query Self-Attention for Sequential RecommendationProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635787(1003-1011)Online publication date: 4-Mar-2024
      • (2022)Visual Planarization in Oil Painting Techniques in Digital Information AgeInternational Transactions on Electrical Energy Systems10.1155/2022/99392092022(1-10)Online publication date: 22-Sep-2022
      • (2022)Fast Nonlinear Image Unblending2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV51458.2022.00325(3193-3201)Online publication date: Jan-2022
      • (2022)Stroke Based Shadow Generation For Line Drawings2022 7th International Conference on Image, Vision and Computing (ICIVC)10.1109/ICIVC55077.2022.9887289(735-740)Online publication date: 26-Jul-2022
      • (2022)Pencil Sketch Generation Based on Stroke Density and TextureInterdisciplinary Research for Printing and Packaging10.1007/978-981-19-1673-1_14(78-85)Online publication date: 17-Apr-2022
      • (2021)Research on Digital Application of Lighting Design in Public Space Based on Cloud Computing and Data MiningWireless Communications and Mobile Computing10.1155/2021/88024582021(1-12)Online publication date: 15-Oct-2021
      • (2021)Generation of Digital Art Composition Using a Multilabel Learning AlgorithmMathematical Problems in Engineering10.1155/2021/34628462021(1-8)Online publication date: 9-Sep-2021
      • (2021)Painting Style-Aware Manga Colorization Based On Generative Adversarial Networks2021 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP42928.2021.9506254(1739-1743)Online publication date: 19-Sep-2021
      • (2021)SmartShadow: Artistic Shadow Drawing Tool for Line Drawings2021 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV48922.2021.00534(5371-5380)Online publication date: Oct-2021
      • (2020)Neural Relighting using Specular Highlight MapJournal of the Korea Computer Graphics Society10.15701/kcgs.2020.26.3.8726:3(87-97)Online publication date: 1-Jul-2020

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media