Skip to main content
Springer Nature Link
Log in

Cross-Domain Learning for Reference-Based Sketch Colorization with Structural and Colorific Strategy

  • Conference paper
  • First Online:
Artificial Neural Networks and Machine Learning – ICANN 2022 (ICANN 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13531))

Included in the following conference series:

  • 2175 Accesses

Abstract

This paper aims to tackle the colorization task of sketch image given an already-colored reference image. Sketch colorization is a thorny task for computer vision since neither grayscale values nor semantic information exists in sketch images. To address this, We propose to jointly train the domain alignment network with a simple adversarial strategy, that we term the structural and colorific conditions, to learn the semantical correspondence between information-scarce sketch and the given instructive reference. Specifically, the inputs from distinct domains will be aligned to an embedding space where the semantical correspondence is established, then, the generator will reconstruct the sketch image according to the established correspondence. We demonstrate the effectiveness of our proposed method in sketch colorization tasks via quantitative and qualitative evaluation against existing approaches in terms of image quality as well as style relevance.

This work supported by the Opening Project of Guangdong Province Key Laboratory of Computational Science at the Sun Yat-Sen. University.2021011, and Guangdong Provincial Department of Education, China: No.PROJ007143460458860544,2019.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bugeau, A., Ta, V.T., Papadakis, N.: Variational exemplar-based image colorization. IEEE Trans. Image Process. 23(1), 298–307 (2013)

    Article  MathSciNet  Google Scholar 

  2. Chen, W., Hays, J.: SketchyGAN: towards diverse and realistic sketch to image synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9416–9425 (2018)

    Google Scholar 

  3. Choi, Y., Uh, Y., Yoo, J., Ha, J.W.: StarGAN v2: diverse image synthesis for multiple domains. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2020)

    Google Scholar 

  4. Ci, Y., Ma, X., Wang, Z., Li, H., Luo, Z.: User-guided deep anime line art colorization with conditional adversarial networks. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 1536–1544 (2018)

    Google Scholar 

  5. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems 27 (2014)

    Google Scholar 

  6. Hati, Y., Jouet, G., Rousseaux, F., Duhart, C.: PaintsTorch: a user-guided anime line art colorization tool with double generator conditional adversarial network. In: European Conference on Visual Media Production, pp. 1–10 (2019)

    Google Scholar 

  7. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Advances in Neural Information Processing Systems 30 (2017)

    Google Scholar 

  8. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

    Google Scholar 

  9. Kim, H., Jhoo, H.Y., Park, E., Yoo, S.: Tag2Pix: line art colorization using text tag with SECat and changing loss. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9056–9065 (2019)

    Google Scholar 

  10. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  11. Lee, J., Kim, E., Lee, Y., Kim, D., Chang, J., Choo, J.: Reference-based sketch image colorization using augmented-self reference and dense semantic correspondence. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5801–5810 (2020)

    Google Scholar 

  12. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)

    Google Scholar 

  13. Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Advances in Neural Information Processing Systems, pp. 700–708 (2017)

    Google Scholar 

  14. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)

    Google Scholar 

  15. Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2337–2346 (2019)

    Google Scholar 

  16. Richardson, E., et al.: Encoding in style: a styleGAN encoder for image-to-image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2287–2296 (2021)

    Google Scholar 

  17. Streijl, R.C., Winkler, S., Hands, D.S.: Mean opinion score (MOS) revisited: methods and applications, limitations and alternatives. Multimedia Syst. 22(2), 213–227 (2016)

    Article  Google Scholar 

  18. Sun, T.H., Lai, C.H., Wong, S.K., Wang, Y.S.: Adversarial colorization of icons based on contour and color conditions. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 683–691 (2019)

    Google Scholar 

  19. Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8798–8807 (2018)

    Google Scholar 

  20. Winnemöller, H., Kyprianidis, J.E., Olsen, S.C.: XDoG: an extended difference-of-gaussians compendium including advanced image stylization. Comput. Graph. 36(6), 740–753 (2012)

    Article  Google Scholar 

  21. Winnemöller, H., Olsen, S.C., Gooch, B.: Real-time video abstraction. ACM Trans. Graph. (TOG) 25(3), 1221–1226 (2006)

    Article  Google Scholar 

  22. Xu, Z., Wang, T., Fang, F., Sheng, Y., Zhang, G.: Stylization-based architecture for fast deep exemplar colorization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9363–9372 (2020)

    Google Scholar 

  23. Yin, W., Lu, P., Zhao, Z., Peng, X.: Yes, “attention is all you need”, for exemplar based colorization. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 2243–2251 (2021)

    Google Scholar 

  24. Yuan, M., Simo-Serra, E.: Line art colorization with concatenated spatial attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3946–3950 (2021)

    Google Scholar 

  25. Zhang, B., et al.: Deep exemplar-based video colorization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8052–8061 (2019)

    Google Scholar 

  26. Zhang, L., Li, C., Simo-Serra, E., Ji, Y., Wong, T.T., Liu, C.: User-guided line art flat filling with split filling mechanism. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9889–9898 (2021)

    Google Scholar 

  27. Zhang, P., Zhang, B., Chen, D., Yuan, L., Wen, F.: Cross-domain correspondence learning for exemplar-based image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5143–5153 (2020)

    Google Scholar 

  28. Zhang, Q., Wang, B., Wen, W., Li, H., Liu, J.: Line art correlation matching feature transfer network for automatic animation colorization. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3872–3881 (2021)

    Google Scholar 

  29. Zhang, R., et al.: Real-time user-guided image colorization with learned deep priors. arXiv preprint arXiv:1705.02999 (2017)

  30. Zhao, M., Zhu, S.C.: Portrait painting using active templates. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Non-photorealistic Animation and Rendering, pp. 117–124 (2011)

    Google Scholar 

  31. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Mathematics and Informatics, South China Agricultural University, Guangzhou, China

    Haowei Zhong, Hailong Liu, Yulu Fu & Jinrong Cui

  2. Animation Department, College of Arts, South China Agricultural University, Guangzhou, China

    Xianzhi Tu

Authors
  1. Haowei Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xianzhi Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hailong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yulu Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jinrong Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinrong Cui .

Editor information

Editors and Affiliations

  1. University of the West of England, Bristol, UK

    Elias Pimenidis

  2. Lancaster University, Lancaster, UK

    Plamen Angelov

  3. Digital Innovation, Teeside University, Middlesbrough, UK

    Chrisina Jayne

  4. Democritus University of Thrace, Xanthi, Greece

    Antonios Papaleonidas

  5. The University of the West of England, Bristol, UK

    Mehmet Aydin

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhong, H., Tu, X., Liu, H., Fu, Y., Cui, J. (2022). Cross-Domain Learning for Reference-Based Sketch Colorization with Structural and Colorific Strategy. In: Pimenidis, E., Angelov, P., Jayne, C., Papaleonidas, A., Aydin, M. (eds) Artificial Neural Networks and Machine Learning – ICANN 2022. ICANN 2022. Lecture Notes in Computer Science, vol 13531. Springer, Cham. https://doi.org/10.1007/978-3-031-15934-3_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-15934-3_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-15933-6

  • Online ISBN: 978-3-031-15934-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics