Skip to main content
SpringerLink
Log in

Color-UNet++: A resolution for colorization of grayscale images using improved UNet++

  • 1166: Advances of machine learning in data analytics and visual information processing
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Colorization is the computer-assisted application of color to a gray scale image, which presents two problems to modern deep learning-based approaches. One is to provide colorization models with both high expressibility and strong learning ability, as current models have difficulty both excelling at coloring and being easy to train. The other is to return a picture without uneven overlap. This paper proposes a deep convolutional network framework called Color-UNet++ for the end-to-end solution of these colorization problems. Color-UNet++ is adjusted to settle gradient dispersion and explosion by capturing more transfer and intermediate results during backpropagation. We adjust the de-convolution structure to solve the problem of uneven overlap. We design the model in YUV instead of RGB color space, with an objective function that is appropriate to the coloring problem and can capture a wide range of colors. A large number of experimental results on LFW and LSUN datasets confirm the method’s superiority.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Bao B, Fu H (2019) Scribble-based colorization for creating smooth-shaded vector graphics. Comput Graph 81:73–81

    Article  Google Scholar 

  2. Billaut V, De Rochemonteix M, Thibault M (2018) Colorunet: A convolutional classification approach to colorization

  3. Bo L, Lai Y-K, John M, Rosin PL (2019) Automatic example-based image colorization using location-aware cross-scale matching. IEEE Trans Image Process 28(9):4606–4619

    Article  MathSciNet  Google Scholar 

  4. Boulkenafet Z, Komulainen J, Hadid A (2018) On the generalization of color texture-based face anti-spoofing. Image Vision Comput 77:1–9

    Article  Google Scholar 

  5. Cao Y, Zhou Z, Zhang W, Yu Y (2017) Unsupervised diverse colorization via generative adversarial networks. CoRR, arXiv:abs/1702.06674

  6. Charpiat G, Hofmann M, Schölkopf B (2008) Automatic image colorization via multimodal predictions. In: Computer vision - ECCV 2008, 10th european conference on computer vision, marseille, france, october 12-18, 2008, proceedings, Part III, pp 126–139

  7. Chen Y, Liu L, Tao J, Xia R, Xi C (2020) The improved image inpainting algorithm via encoder and similarity constraint. The Visual Computer (3)

  8. Cheng Z, Yang Q, Sheng B (2015) Deep colorization. In: 2015 IEEE International conference on computer vision, ICCV 2015, santiago, chile, december 7-13, 2015, pp 415–423

  9. Deshpande A, Rock J, Forsyth DA (2015) Learning large-scale automatic image colorization. In: 2015 IEEE International conference on computer vision, ICCV 2015, santiago, chile, december 7-13, 2015, pp 567–575

  10. Fang F, Wang T, Zeng T, Zhang G (2020) A superpixel-based variational model for image colorization. IEEE Trans Vis Comput Graph 26(10):2931–2943

    Article  Google Scholar 

  11. Fisher Y u, Zhang Yinda, Song Shuran, Seff Ari, Xiao Jianxiong (2015) LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. CoRR, arXiv:abs/1506.03365

  12. Huang GB, Mattar MA, Lee H, Learned-miller EG (2012) Learning to align from scratch. In: Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pp 773–781

  13. Iizuka Satoshi, Simo-Serra E, Ishikawa H (2016) Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans Graph 35(4):110,1–110,11

    Article  Google Scholar 

  14. Irony R, Cohen-Or D, Lischinski D (2005) Colorization by Example. In: Bala K, Dutre P (eds) Eurographics Symposium on Rendering (2005). The Eurographics Association

  15. Isola P, Zhu J-Y, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on computer vision and pattern recognition, CVPR 2017, honolulu, HI, USA, July 21-26, 2017, pp 5967–5976

  16. Kuzovkin D, Chamaret C, Pouli T (2015) Descriptor-based image colorization and regularization. In: Computational color imaging - 5th international workshop, CCIW 2015, saint etienne, france, march 24-26, 2015, proceedings, pp 59–68

  17. Limmer M, Lensch HPA (2016) Infrared colorization using deep convolutional neural networks. In: 15Th IEEE international conference on machine learning and applications, ICMLA 2016, anaheim, CA, USA, December 18-20, 2016, pp 61–68

  18. Larsson G, Maire M, Shakhnarovich G (2016) Learning representations for automatic colorization. In: Computer vision - ECCV 2016 - 14th european conference, amsterdam, the netherlands, october 11-14, 2016, proceedings, Part IV, pp 577–593

  19. Learned-Miller E, Huang GB, RoyChowdhury A, Li H, Hua G (2016) Labeled faces in the wild: a survey. In: Advances in face detection and facial image analysis, Springer, pp 189–248

  20. Levin A, Lischinski D, Weiss Y (2004) Colorization using optimization. ACM Trans Graph 23(3):689–694

    Article  Google Scholar 

  21. Li P, Xu J, Mou J, Yang F (2019) Fractional-order 4d hyperchaotic memristive system and application in color image encryption. EURASIP J Image Video Process 2019:22

    Article  Google Scholar 

  22. Luan Qing, Wen Fang, Cohen-or D, Liang L, Xu Y-Q, Shum H-Y (2007) Natural image colorization. In: Proceedings of the eurographics symposium on rendering techniques, Grenoble, France, 2007, pp 309–320

  23. Morimoto Y, Taguchi Y, Naemura T (2009) Automatic colorization of grayscale images using multiple images on the Web. In: International conference on computer graphics and interactive techniques, SIGGRAPH 2009, new orleans, louisiana, USA, August 3-7, 2009, Poster Proceedings

  24. Nazeri K, Ng E, Ebrahimi M (2018) Image colorization using generative adversarial networks. In: Articulated motion and deformable objects - 10th international conference, AMDO 2018, palma de mallorca, spain, july 12-13, 2018, proceedings, pp 85–94

  25. Nguyen V, Sintunata V, Aoki T (2016) Automatic image colorization based on feature lines

  26. Odena A, Dumoulin V, Olah C (2016) Deconvolution and checkerboard artifacts. Distill

  27. Roohi S, Forouzandeh A (2019) Regarding color psychology principles in adventure games to enhance the sense of immersion. Entertainment Computing 30

  28. Varga D, Szirányi T (2017) Twin deep convolutional neural network for example-based image colorization. In: Computer analysis of images and patterns - 17th international conference, CAIP 2017, ystad, sweden, august 22-24, 2017, proceedings, Part I, pp 184–195

  29. Xia Z, Wang X, Wang M, Unar S, Wang C, Liu Y, Li X (2019) Geometrically invariant color medical image null-watermarking based on precise quaternion polar harmonic fourier moments, vol 7

  30. Yatziv L, Sapiro G (2006) Fast image and video colorization using chrominance blending. IEEE Trans Image Process 15(5):1120–1129

    Article  Google Scholar 

  31. Yu F, Li L, Xiao L, Li K, Cai S (2019) A robust and fixed-time zeroing neural dynamics for computing time-variant nonlinear equation using a novel nonlinear activation function. Neurocomputing 350:108–116

    Article  Google Scholar 

  32. Zhang R, Isola P, Efros AA (2016) Colorful image colorization. In: Computer vision - ECCV 2016 - 14th european conference, amsterdam, the netherlands, october 11-14, 2016, proceedings, Part III, pp 649–666

  33. Zhang R, Isola P, Efros AA (2017) Colorize photos https://demos.algorithmia.com/colorize-photos/

  34. Zhang J, Xie Z, Sun J, Zou X, Wang J (2020) A cascaded R-CNN with multiscale attention and imbalanced samples for traffic sign detection. IEEE Access 8:29742–29754

    Article  Google Scholar 

  35. Zheng S, Wang L, Ling B, Hu D (2017) Coverless information hiding based on robust image hashing. In: International conference on intelligent computing

  36. Zhou Z, Siddiquee MdMR, Tajbakhsh N, Liang J (2018) Unet++: a nested u-net architecture for medical image segmentation. CoRR, arXiv:abs/1807.10165

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 61762089, Grant 61663047, Grant 61863036, Grant 61762092, and 62002313, Key Areas Research Program of Yunnan Province under Grant No.202001BB050076 and the Yunnan Applied Basic Research Projects under Grant No.202001BB050034. The Open Foundation of Key Laboratory in Software Engineering of Yunnan Province under Grant No.2020SE307 and No. 2020SE408.

Author information

Authors and Affiliations

  1. National Pilot School of Software, Yunnan University, Yunnan, China

    Yide Di, Xiaoke Zhu, Xin Jin, Qiwei Dou, Wei Zhou & Qing Duan

  2. Engineering Research Center of Cyberspace, Yunnan University, Yunnan, China

    Xin Jin, Wei Zhou & Qing Duan

Authors
  1. Yide Di
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoke Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qiwei Dou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wei Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qing Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qing Duan.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Di, Y., Zhu, X., Jin, X. et al. Color-UNet++: A resolution for colorization of grayscale images using improved UNet++. Multimed Tools Appl 80, 35629–35648 (2021). https://doi.org/10.1007/s11042-021-10830-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10830-2

Keywords

Search

Navigation