Skip to main content
SpringerLink
Log in

Hand-drawn grayscale image colorful colorization based on natural image

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

This paper presents a novel colorization technique for hand-drawn grayscale images, such as cartoons and sketches, based on reference natural images and simple user interactions, which can generate colorful and natural results. Firstly, to solve the problem of inputting complex color scribbles by simple interactions, we introduce a region thinning method for generating color scribbles and then map these scribbles to the corresponding regions in the hand-drawn image via region mapping method based on the scanning line. Secondly, to maintain the color fidelity, a luminance downward modification method is proposed. Next, a optimization method for colorization is proposed, where the feature vectors are adjusted by a smooth feature map, thus maintaining smooth color transitions in the smooth areas and controlling the color overflow in the texture areas. Thirdly, to fuse the colorized image with a highlight effect, a luminance upward modification method by weights, which are determined by color distance and boundary distance, is proposed. The experimental results of the proposed algorithm show the smooth color transition in the intensity-continuity areas, color overflow controlling in the texture-continuity areas and natural effect of light and shadow.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Notes

  1. https://demos.algorithmia.com/colorize-photos/.

  2. https://github.com/DwangoMediaVillage/Comicolorization.

References

  1. Welsh, T., Ashikhmin, M., Mueller, K.: Transferring color to greyscale images. Acm Trans. Graph. 21(3), 277–280 (2002)

    Article  Google Scholar 

  2. Liu, S., Zhang, X.: Automatic grayscale image colorization using histogram regression. Pattern Recognit. Lett. 33(13), 1673–1681 (2012)

    Article  Google Scholar 

  3. Gauge, C., Sasi, S.: Automated colorization of grayscale images using texture descriptors and a modified fuzzy c-means clustering. J. Intell. Learn. Syst. Appl. 4(2), 135–143 (2012)

    Google Scholar 

  4. Liu, S., Zhang, X.: Image colorization based on texture map. J. Electron. Imaging 22(1), 3011 (2013)

    Article  Google Scholar 

  5. Arbelot, B., Hurtut, T., Hurtut, T., Thollot, J.: Automatic texture guided color transfer and colorization. In: Joint Symposium on Computational Aesthetics and Sketch Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering, pp. 21–32 (2016)

  6. Chen, T., Wang, Y., Schillings, V., Meinel, C.: Grayscale image matting and colorization. In: Proceedings of Asian Conference on Computer Vision, vol. 6. Citeseer (2004)

  7. Irony, R., Cohen-Or, D., Lischinski, D.: Colorization by example. In: Eurographics Symposium on Rendering Techniques, Konstanz, Germany, June 29–July, pp. 201–210 (2005)

  8. Schnitman, Y., Caspi, Y., Cohen-Or, D., Lischinski, D.: Inducing semantic segmentation from an example. In: Asian Conference on Computer Vision, pp. 373–384 (2006)

  9. Li, B., Zhao, F., Su, Z., Liang, X., Lai, Y.K., Rosin, P.L.: Example-based image colorization using locality consistent sparse representation. IEEE Trans. Image Process. 26(11), 5188–5202 (2017)

    Article  MathSciNet  Google Scholar 

  10. Charpiat, G., Bezrukov, I., Hofmann, M., Altun, Y., Schlkopf, B., Lukac, R.: Machine learning methods for automatic image colorization. Ann. Stat. 36(3), 1171–1220 (2011)

    Google Scholar 

  11. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. 35(4), 1–11 (2016)

    Article  Google Scholar 

  12. Zhang, R., Isola, P., Efros, A.A.: Colorful Image Colorization. Springer, Berlin (2016)

    Book  Google Scholar 

  13. Luo, X., Luo, X., Luo, X., Luo, X., Luo, X.: Deep patch-wise colorization model for grayscale images. In: SIGGRAPH ASIA 2016 Technical Briefs, p. 13 (2016)

  14. Furusawa, C., Hiroshiba, K., Ogaki, K., Odagiri, Y.: Comicolorization: semi-automatic manga colorization. In: SIGGRAPH Asia 2017 Technical Briefs, p. 12. ACM (2017)

  15. Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. In: ACM SIGGRAPH, pp. 689–694 (2004)

  16. Kawulok, M., Smolka, B.: Competitive image colorization. In: IEEE International Conference on Image Processing, pp. 405–408 (2010)

  17. Koo, H.I., Cho, N.I.: Colorization based on soft segmentation. J. Electron. Imaging 20(1), 010502–010502-3 (2011)

    Article  Google Scholar 

  18. Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc. 15(5), 1120–1129 (2006)

    Article  Google Scholar 

  19. Qu, Y., Wong, T.T., Heng, P.A.: Manga colorization. ACM Trans. Graph. (SIGGRAPH 2006 issue) 25(3), 1214–1220 (2006)

    Article  Google Scholar 

  20. Luan, Q., Wen, F., Cohen-Or, D., Liang, L., Xu, Y.Q., Shum, H.Y.: Natural image colorization. In: Eurographics Conference on Rendering Techniques, pp. 309–320 (2007)

  21. Sheng, B., Sun, H., Chen, S., Liu, X., Wu, E.: Colorization using the rotation-invariant feature space. IEEE Comput. Graph. Appl. 31(2), 24–35 (2011)

    Article  Google Scholar 

  22. Casaca, W., Colnago, M., Nonato, L.G.: Interactive image colorization using Laplacian coordinates. In: International Conference on Computer Analysis of Images and Patterns, pp. 675–686. Springer (2015)

  23. Sykora, D., Dingliana, J., Collins, S.: Lazybrush: flexible painting tool for hand-drawn cartoons. Comput. Graph. Forum 28(2), 599–608 (2010)

    Article  Google Scholar 

  24. Sýkora, D., Ben-Chen, M., Whited, B., Simmons, M.: Textoons: practical texture mapping for hand-drawn cartoon animations. In: International Symposium on Non-Photorealistic Animation and Rendering 2009, Vancouver, pp. 75–84 (2011)

  25. Sangkloy, P., Lu, J., Fang, C., Yu, F., Hays, J.: Scribbler: controlling deep image synthesis with sketch and color. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6836–6845 (2017)

  26. Ding, X., Xu, Y., Deng, L., Yang, X.: Colorization using quaternion algebra with automatic scribble generation. In: Advances in Multimedia Modeling-International Conference, MMM 2012, Klagenfurt, Austria, January 4–6, 2012. Proceedings, pp. 103–114 (2012)

  27. Rusu, C., Tsaftaris, S.A.: Estimation of scribble placement for painting colorization. In: International Symposium on Image and Signal Processing and Analysis, pp. 564–569 (2013)

  28. Rother, C., Kolmogorov, V., Blake, A.: ”Grabcut”: interactive foreground extraction using iterated graph cuts. In: ACM SIGGRAPH, pp. 309–314 (2004)

  29. Wegner, S., Harms, T., Oswald, H., Fleck, E.: The watershed transformation on graphs for the segmentation of ct images. In: International Conference on Pattern Recognition, vol. 3, pp. 498–502 (1996)

  30. Ge, Y., Fitzpatrick, J.M.: On the generation of skeletons from discrete euclidean distance maps. IEEE Trans. Pattern Anal. Mach. Intell. 18(11), 1055–1066 (1996)

    Article  Google Scholar 

  31. Wu, E., Liu, F.: Robust image metamorphosis immune from ghost and blur. Visual Comput. 29(4), 311–321 (2013)

    Article  Google Scholar 

  32. Fan, X., Feng, Y., Chai, Z., Gu, X.D., Luo, Z.: Image morphing with conformal welding. Vis. Comput. 32(9), 1191–1203 (2016)

    Article  Google Scholar 

Download references

Funding

This study is funded by the National Natural Science Foundation of China (51775492) and China Postdoctoral Science Foundation (2018M630670).

Author information

Authors and Affiliations

  1. State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, China

    Liyang Fang, Jin Wang, Guodong Lu & Jianhui Fu

  2. International Design Institute, Zhejiang University, Hangzhou, China

    Dongliang Zhang

Authors
  1. Liyang Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guodong Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dongliang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jianhui Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jin Wang.

Ethics declarations

Conflict of interest

All the authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fang, L., Wang, J., Lu, G. et al. Hand-drawn grayscale image colorful colorization based on natural image. Vis Comput 35, 1667–1681 (2019). https://doi.org/10.1007/s00371-018-1613-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-018-1613-8

Keywords

Search

Navigation