Abstract
Semi-realistic paintings are widely used in game concept art, film posters and animation, etc. However, none of existing work can reproduce this type of art. We propose the SemiPainter to generate semi-realistic paintings from line drawings based on a deep learning network. We divide the complex work into two parts, generating the shadow from a line drawing and a simple flat shadow image at first, then generating flat color from line drawings. We merge these two paintings and get the final result. We use two networks with U-Net structure to realize these two stages of work. In order to ensure the global consistency of picture illumination, we add the lighting direction to the shadow network. The color of each part of the painting is uncertain, so color hints from users is provided when coloring, so that the network can realize controllable colorization. To the best of our knowledge, this is the first framework for reproducing semi-realistic paintings.
Keywords
This work was partly supported by the Natural Science Foundation of China under grant no. 62072328.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bahng, H., et al.: Coloring with words: guiding image colorization through text-based palette generation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11216, pp. 443–459. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01258-8_27
Chen, S.Y., et al.: Active colorization for cartoon line drawings. IEEE Trans. Vis. Comput. Graph. 28(2), 1198–1208 (2022)
Chen, Y., Lai, Y.K., Liu, Y.J.: CartoonGAN: generative adversarial networks for photo cartoonization. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9465–9474 (2018)
Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: IEEE International Conference on Computer Vision (ICCV), pp. 415–423 (2015)
Deshpande, A., Lu, J., Yeh, M.C., Chong, M.J., Forsyth, D.: Learning diverse image colorization. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2877–2885 (2017)
Fang, F., Wang, T., Zeng, T., Zhang, G.: A superpixel-based variational model for image colorization. IEEE Trans. Vis. Comput. Graph. 26(10), 2931–2943 (2020)
Furusawa, C., Hiroshiba, K., Ogaki, K., Odagiri, Y.: Comicolorization: semi-automatic manga colorization. In: SIGGRAPH Asia Technical Briefs, pp. 1–4 (2017)
Gao, Z., onetsuji, T.Y., Takamura, T., Matsuoka, T., Naradowsky, J.: Automatic illumination effects for 2D characters. In: NIPS Workshop on Machine Learning for Creativity and Design (2018)
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414–2423 (2016)
He, M., Chen, D., Liao, J., Sander, P.V., Yuan, L.: Deep exemplar-based colorization. ACM Trans. Graph. 37(4), 1–16 (2018)
Hensman, P., Aizawa, K.: cGAN-based manga colorization using a single training image. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 03, pp. 72–77 (2017)
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1510–1519 (2017)
Huang, Z., Zhou, S., Heng, W.: Learning to paint with model-based deep reinforcement learning. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8708–8717 (2019)
Hudon, M., Grogan, M., Pagés, R., Smolić, A.: Deep normal estimation for automatic shading of hand-drawn characters. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11131, pp. 246–262. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11015-4_20
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976 (2017)
Jin, Y., Li, P., Sheng, B., Nie, Y., Kim, J., Wu, E.: SRNPD: spatial rendering network for pencil drawing stylization. Comput. Animation Virtual Worlds 30(3–4), e1890 (2019)
Jin, Y., Sheng, B., Li, P., Chen, C.L.P.: Broad colorization. IEEE Trans. Neural Netw. Learn. Syst. 32(6), 2330–2343 (2021)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Kolkin, N., Salavon, J., Shakhnarovich, G.: Style transfer by relaxed optimal transport and self-similarity. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10043–10052 (2019)
Li, C., Wand, M.: Precomputed real-time texture synthesis with Markovian generative adversarial networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 702–716. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_43
Li, J., Gao, N., Shen, T., Zhang, W., Mei, T., Ren, H.: SketchMan: learning to create professional sketches, pp. 3237–3245. Association for Computing Machinery, New York (2020)
Liao, J., Yao, Y., Yuan, L., Hua, G., Kang, S.B.: Visual attribute transfer through deep image analogy. ACM Trans. Graph. 36(4), 1–15 (2017)
Lin, D., Wang, Y., Xu, G., Li, J., Fu, K.: Transform a simple sketch to a Chinese painting by a multiscale deep neural network. Algorithms 11(1), 1–18 (2018)
Liu, S.: Two decades of colorization and decolorization for images and videos. arXiv: 2204.13322 (2022)
Liu, S., Zhang, X.: Automatic grayscale image colorization using histogram regression. Pattern Recogn. Lett. 33(13), 1673–1681 (2012)
Liu, S., Zhang, X.: Image colorization based on texture map. J. Electron. Imaging 22(1), 1–9 (2013)
Liu, S., Zhu, T.: Structure-guided arbitrary style transfer for artistic image and video. IEEE Trans. Multimed. 24, 1299–1312 (2022)
Liu, S., et al.: Paint transformer: feed forward neural painting with stroke prediction. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6578–6587 (2021)
Liu, Y., Qin, Z., Wan, T., Luo, Z.: Auto-painter: cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks. Neurocomputing 311, 78–87 (2018)
Park, D.Y., Lee, K.H.: Arbitrary style transfer with style-attentional networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5873–5881 (2019)
Sangkloy, P., Lu, J., Fang, C., Yu, F., Hays, J.: Scribbler: controlling deep image synthesis with sketch and color. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6836–6845 (2017)
Sheng, B., Li, P., Gao, C., Ma, K.L.: Deep neural representation guided face sketch synthesis. IEEE Trans. Vis. Comput. Graph. 25(12), 3216–3230 (2019)
Su, W., Du, D., Yang, X., Zhou, S., Fu, H.: Interactive sketch-based normal map generation with deep neural networks. Proc. ACM Comput. Graph. Interact. Tech. 1(1), 1–17 (2018)
Xiao, C., et al.: Example-based colourization via dense encoding pyramids. Comput. Graph. Forum 39(1), 20–33 (2020)
Xue, A.: End-to-end Chinese landscape painting creation using generative adversarial networks. arXiv: 2011.05552 (2020)
Ye, R.T., Wang, W.L., Chen, J.C., Lin, K.W.: Interactive anime sketch colorization with style consistency via a deep residual neural network. In: 2019 International Conference on Technologies and Applications of Artificial Intelligence (TAAI), pp. 1–5 (2019)
Zan, T.: Paintschainer (2017). https://petalica-paint.pixiv.dev/index_zh
Zhang, L., Jiang, J., Ji, Y., Liu, C.: SmartShadow: artistic shadow drawing tool for line drawings. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5371–5380 (2021)
Zhang, L., Li, C., Wong, T.T., Ji, Y., Liu, C.: Two-stage sketch colorization. ACM Trans. Graph. 37(6), 1–14 (2018)
Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40
Zhang, R., et al.: Real-time user-guided image colorization with learned deep priors. ACM Trans. Graph. 36(4), 1–11 (2017)
Zheng, Q., Li, Z., Bargteil, A.: Learning to shadow hand-drawn sketches. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7434–7443 (2020)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251 (2017)
Zou, Z., Shi, T., Qiu, S., Yuan, Y., Shi, Z.: Stylized neural painting. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15684–15693 (2021)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Fan, K., Liu, S., Lu, W. (2022). SemiPainter: Learning to Draw Semi-realistic Paintings from the Manga Line Drawings and Flat Shadow. In: Magnenat-Thalmann, N., et al. Advances in Computer Graphics. CGI 2022. Lecture Notes in Computer Science, vol 13443. Springer, Cham. https://doi.org/10.1007/978-3-031-23473-6_24
Download citation
DOI: https://doi.org/10.1007/978-3-031-23473-6_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23472-9
Online ISBN: 978-3-031-23473-6
eBook Packages: Computer ScienceComputer Science (R0)