Skip to main content
SpringerLink
Log in

A fully-automatic image colorization scheme using improved CycleGAN with skip connections

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Image colorization is the process of assigning different RGB values to each pixel of a given grayscale image to obtain the corresponding colorized image. In this work, we propose a new automatic image colorization method based on the modified cycle-consistent generative adversarial network (CycleGAN). This method can generate a natural color image with only one given gray image without reference image or manual interaction. In the proposed method, we first modify the original network structure by combining a u-shaped network with a skip connection to improve the ability of feature representation in image colorization. Meanwhile, we design a compounded loss function to measure the errors between the ground-truth image and the predicted result to improve the authenticity and naturalness of the colorized image; further, we also add the detail loss function to ensure that the details of the generated color and grayscale images are substantially similar. Finally, the performance of the proposed model is verified on different datasets. Experiments show that our method can generate more realistic color images when compared to other methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26

Similar content being viewed by others

References

  1. Attique M, Gilanie G, Ullah H et al (2012) Colorization and Automated Segmentation of Human T2 MR Brain Images for Characterization of Soft Tissues. PLOS ONE 7:e33616

    Article  Google Scholar 

  2. Aquino G, De Jesus Rubio J, Pacheco J et al (2020) Novel Nonlinear Hypothesis for the Delta Parallel Robot Modeling. IEEE Access 8(1):46324–46334

    Article  Google Scholar 

  3. Bahng H, Yoo S, Cho W et al, Bahng H et al (2018) Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation., European Conference on Computer Vision (ECCV), Munich

  4. Bi Z, Yu L, Gao H et al (2020) Improved VGG model-based efficient traffic sign recognition for safe driving in 5G scenarios International Journal of Machine Learning and Cybernetics

  5. Chai C, Liao J, Zou N et al (2018) A one-to-many conditional generative adversarial network framework for multiple image-to-image translations. Multimed Tools Appl 77:22339–22366. https://doi.org/10.1007/s11042-018-5968-7

    Article  Google Scholar 

  6. Cao Y, Zhou Z, Zhang W et al (2017) Unsupervised Diverse Colorization via Generative Adversarial Networks. Lect Notes Comput Sci (LNCS) 10534:151–166

    Article  Google Scholar 

  7. Chang HW, Fried O, Liu YM et al (2015) Palette-based Photo Recoloring. ACM Trans Graph 34(139):10017. New York

  8. Chen J, Ying H, Liu X, Gu J, Feng R, Chen T, Wu JA (2020) Transfer Learning Based Super-Resolution Microscopy for Biopsy Slice Images: The Joint Methods Perspective. IEEE/ACM Transactions on Computational Biology and Bioinformatics

  9. Cheng ZZ, Yang Q, Sheng B (2015) Deep Colorization. IEEE International Conference on Computer Vision (ICCV), pp 415–423

  10. Chiang HS, Chen MY, Huang YJ (2019) Wavelet-Based EEG Processing for Epilepsy Detection Using Fuzzy Entropy and Associative Petri Net. IEEE Access 7:103255-103262

  11. De Jesus R (2009) J SOFMLS: online self-organizing fuzzy modified least-squares network. IEEE Trans Fuzzy Syst 17(6):1296–1309

    Article  Google Scholar 

  12. Elias I, De J, de Jesus Rubio J et al (2020) Genetic Algorithm with Radial Basis Mapping Network for the Electricity Consumption Modeling. Appl Sci 10(12):4239

    Article  Google Scholar 

  13. Fan GF, Qing S, Wang H et al (2013) Support vector regression model based on empirical mode decomposition and auto regression for electric load forecasting. Energies 6(4):1887–1901

    Article  Google Scholar 

  14. Fan GF, Peng LL, Hong WC et al (2016) Electric load forecasting by the SVR model with differential empirical mode decomposition and auto regression. Neurocomputing 173:958–970

    Article  Google Scholar 

  15. Fan GF, Wei X, Li YT et al (2020) Forecasting electricity consumption using a novel hybrid model. Sustain Cit Soc 61:102320

    Article  Google Scholar 

  16. Fan GF, Guo YH, Zheng JM et al (2020) A generalized regression model based on hybrid empirical mode decomposition and support vector regression with back propagation neural network for mid-short term load forecasting. J Forecast 39(5):737–756

    Article  MathSciNet  Google Scholar 

  17. Fatima A, Hussain W, Rasool S (2020) Grey is the new RGB: How good is GAN-based image colorization for image compression? Multimed Tools Appl

  18. Fang L, Wang J, Lu G et al (2019) Hand-drawn grayscale image colorful colorization based on natural image. Vis Comput 35:1667–1681

    Article  Google Scholar 

  19. Fang FM, Wang TT, Zeng TY et al (2019) A Superpixel-based Variational Model for Image Colorization IEEE Transactions on Visualization and Computer Graphics (Early Access)

  20. Furusawa C, Hiroshiba K, Ogaki K et al (2017) Comicolorization: Semi-Automatic Manga Colorization. ACM SIGGRAPH Asia

  21. Goodfellow IJ, Pouget-Abadie J, Mirza M et al (2014) Generative adversarial nets. International Conference on Neural Information Processing Systems, pp 2672–2680

  22. Hettiarachchi R, Peters JF (2017) Vorono? region-based adaptive unsupervised color image segmentation. Pattern Recogn 65:119–135

    Article  Google Scholar 

  23. He MC, Gu XJ, Gu XS (2014) A Fast Colorization Algorithm for Infrared Video. Commun Comput Inf Sci 462:282–292

    Google Scholar 

  24. He M, Chen D, Liao J et al (2018) Deep exemplar-based colorization. ACM Transactions on Graphics

  25. Hernandez G, Zamora E, Sossa H et al (2020) Hybrid neural networks for big data classification. Neurocomputing 390:327–340

    Article  Google Scholar 

  26. Ioan V, Lacramioara ST, Mihaela CV (2015) Client-side Medical Image Colorization in a Collaborative Environment. Stud Health Technol Inf 210:904–908

    Google Scholar 

  27. Isola P, Zhu JY, Zhou T et al (2017) Image-to-Image Translation with Conditional Adversarial Networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1125–1134

  28. Iizuka S, Simo-Serra E, Ishikawa H (2016) Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans Graph 35:4

    Article  Google Scholar 

  29. Iizuka SA, Simoserra E (2019) DeepRemaster: Temporal Source−Reference Attention Networks for Comprehensive Video Enhancement. SIGGRAPH Asia, BCEC, Brisbane

  30. Johari MM, Behroozi H (2020) Context-aware colorization of gray-scale images utilizing a cycle-consistent generative adversarial network architecture. Neurocomputing 407:94–104

    Article  Google Scholar 

  31. Larsson G, Maire M, Shakhnarovich G (2016) Learning Representations for Automatic Colorization. European Conference on Computer Vision (ECCV), pp 577–593

  32. Lei CY, Chen QF (2019) Fully Automatic Video Colorization with Self-Regularization and Diversity. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach

  33. Lin B, Deng S, Gao H, Yin J (2020) A Multi-Scale Activity Transition Network for Data Translation in EEG Signals Decoding. IEEE/ACM Transactions on Computational Biology and Bioinformatics

  34. Lin J, Song X, Gan T et al (2020) PaintNet: A shape-constrained generative framework for generating clothing from fashion model. Multimed Tools Appl

  35. Liu H, Fu ZL, Han JG et al (2018) Single satellite imagery simultaneous super-resolution and colorization using multi-task deep neural networks. J Vis Commun Image Represent 53:20–30

    Article  Google Scholar 

  36. Liu SF, Zhong GY, Mello SD et al (2018) Switchable Temporal Propagation Network. European Conference on Computer Vision (ECCV), pp 87–102

  37. Mathieu G, Luiz Gustavo R, Gregoire M (2019) Analogue based colorization of remote sensing images using textural information. ISPRS J Photogramm Remote Sens 147:242–254

    Article  Google Scholar 

  38. Meda-Campa?a JA (2018) On the Estimation and Control of Nonlinear Systems With Parametric Uncertainties and Noisy Outputs. IEEE Access 6:31968–31973

  39. Messaou S, Forsyth D, Schwing AG (2018) Structural Consistency and Controllability for Diverse Colorization European Conference on Computer Vision (ECCV)

  40. Patricia LS, Angel DS, Boris XV et al (2018) Near InfraRed Imagery Colorization. IEEE International Conference on Image Processing (ICIP), Athens, pp 2237–2241

  41. Sangkloy P, Lu J, Fang C et al (2017) Scribbler: Controlling Deep Image Synthesis with Sketch and Color. IEEE Conf Comput Vis Pattern Recogn (CVPR) 1:6836–6845

    Google Scholar 

  42. Suarez PL, Sappa AD, Vintimilla BX (2017) Infrared Image Colorization Based on a Triplet DCGAN Architecture. Comput Vis Pattern Recogn Workshops 1:212–217

    Google Scholar 

  43. Suarez PL, Sappa AD, Vintimilla BX (2017) Learning to Colorize Infrared Images. International Conference on Practical Applications of Agents and Multi-Agent Systems (PAAMS), pp 164–172

  44. Sykora D, Dingliana J, Collins S (2010) Lazybrush: Flexible painting tool for hand-drawn cartoons. Comput Graph Forum 28(2):599–608

    Article  Google Scholar 

  45. Tylecek R, Sara R (2013) Spatial Pattern Templates for Recognition of Objects with Regular Structure Pattern Recognition. Springer, Berlin

  46. Vondrick C, Shrivastava A, Fathi A et al (2018) Tracking Emerges by Colorizing Videos. European Conference on Computer Vision (ECCV), pp 391–408

  47. Wu M, Jin X, Jiang Q et al (2020) Remote sensing image colorization using symmetrical multi-scale DCGAN in YUV color space. Vis Comput

  48. Wu XD, Hoi SCH (2020) Recent advances in deep learning for object detection. Neurocomputing 396:39–64

    Article  Google Scholar 

  49. Xian W, Sangkloy P, Agrawal V et al (2018) TextureGAN: Controlling deep image synthesis with texture patches. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 8456–8465

  50. Xiao C, Han C, Zhang Z et al (2019) Example-based Colourization via Dense Encoding Pyramids Computer Graphics Forum

  51. Yi X, Zhou P, Zheng Y (2019) Interactive Deep Colorization Using Simultaneous Global and Local Inputs. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

  52. Zhang R, Isola P, Efros AA (2016) Colorful Image Colorization. European Conference on Computer Vision (ECCV), pp 649–666

  53. Zhang R, Zhu JY, Isola P et al (2017) Real-time user-guided image colorization with learned deep priors. ACM Trans Graph 36:119

    Google Scholar 

  54. Zhang LM, Ji Y, Lin X (2017) Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN. Asian Conference on Pattern Recognition (ACPR)

  55. Zhang W, Fang CW, Li GB (2017) Automatic Colorization with Improved Spatial Coherence and Boundary Localization. J Comput Sci Technol 32 (3):494–506

    Article  Google Scholar 

  56. Zhang L, Li C, Wong TT et al (2018) Two-stage sketch colorization. ACM Transactions on Graphics

  57. Zhang B, He MM, Liao J et al (2019) Deep Exemplar−based Video Colorization. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach

  58. Zou X, Wang Z, Li Q et al (2019) Integration of residual network and convolutional neural network along with various activation functions and global pooling for time series classification. Neurocomputing 367:39–45

    Article  Google Scholar 

  59. Zhang Z, Hong WC (2019) Electric load forecasting by complete ensemble empirical model decomposition adaptive noise and support vector regression with quantum-based dragonfly algorithm. Nonlinear Dyn 98:1107–1136

    Article  Google Scholar 

  60. Zhang Z, Ding S, Sun Y (2020) A support vector regression model hybridized with chaotic krill herd algorithm and empirical mode decomposition for regression task. Neurocomputing 410:185–201

    Article  Google Scholar 

  61. Zhu JY, Park T, Isola P et al (2017) Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. IEEE International Conference on Computer Vision (ICCV), pp 2242–2251

Download references

Acknowledgements

This research was funded by the National Natural Science Foundation of China (No. 62002313, 61863036), China Postdoctoral Science Foundation (2020T130564, 2019M653507), Key Areas Research Program of Yunnan Province in China (202001BB050076), the Open Foundation of Key Laboratory in Software Engineering of Yunnan Province under Grant No. 2020SE408 and Postdoctoral Science Foundation of Yunnan Province in China.

Author information

Authors and Affiliations

  1. School of Software, Yunnan University, Kunming, Yunnan, China

    Shanshan Huang, Xin Jin, Qian Jiang, Jie Li, Puming Wang & Shaowen Yao

  2. Engineering Research Center of Cyberspace, Yunnan University, Yunnan, China

    Xin Jin, Qian Jiang, Jie Li, Puming Wang & Shaowen Yao

  3. Institute of Technology Management, National Chiao Tung University, Hsinchu, Taiwan

    Shin-Jye Lee

Authors
  1. Shanshan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qian Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jie Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shin-Jye Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Puming Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Shaowen Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Jin.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, S., Jin, X., Jiang, Q. et al. A fully-automatic image colorization scheme using improved CycleGAN with skip connections. Multimed Tools Appl 80, 26465–26492 (2021). https://doi.org/10.1007/s11042-021-10881-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10881-5

Keywords

Search

Navigation