Skip to main content
SpringerLink
Log in

Image colorization using deep convolutional auto-encoder with multi-skip connections

  • Application of soft computing
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

The colorization of grayscale images is a challenging task in image processing. Recently, deep learning has shown remarkable performance in image colorization. However, the detail loss and color distortion are still serious problem for most existing methods, and some useful features may be lost in the processes of various convolutional layers because of the vanishing gradient problem. Therefore, there is still a considerable space to reach the roof of image colorization. In this work, we propose a deep convolutional auto-encoder with special multi-skip connections for image colorization in YUV color space, and the specific contributions or designs of this work are shown as the following five points. First, a given gray image is used as the Y channel to input a deep learning model to predict U and V channel. Second, the adopted encoder-decoder consists of a main path and two branch paths, and the branch path has two skip connection ways that include one shortcut in each three layers and one shortcut in each six layers. Third, the convolutional kernel size is set as 2*2 that is a special consideration in the path of one shortcut in each six layers. Fourth, a composite loss function is proposed based on the mean square error and gradient that is defined to calculate the errors between the ground truth and the predicted result. Finally, we also discuss the reasonable network parameters, such as the way of shortcut connection, the convolutional kernel size of shortcut connection, and loss function parameters. Experiments on different image datasets show that the proposed image colorization model is effective, and the scores of the PNSR, RMSE, SSIM, and Pearson correlation coefficient are, respectively, to 27.0595, 0.1311, 0.561, and 0.9771.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

Enquiries about data availability should be directed to the authors.

References

  • Abo-Hammour Z, Arqub OA, Alsmadi O et al (2014a) An optimization algorithm for solving systems of second-order singular boundary value problems. Appl Math Information Sci 8(6):2809–2821

    Article  MathSciNet  Google Scholar 

  • Abo-Hammour Z, Arqub OA, Momani S et al (2014b) Optimization solution of Troesch's and Bratu's problems of ordinary type using novel continuous genetic algorithm Discrete Dynamics in Nature and Society, 401696

  • An JC, Kpeyiton KG, Shi Q (2020) Grayscale images colorization with convolutional neural networks. Soft Comput 24(7):4751–4758

    Article  Google Scholar 

  • Arqub OA, Abo-Hammour Z (2014) Numerical solution of systems of second-order boundary value problems using continuous genetic algorithm. Inf Sci 279:396–415

    Article  MathSciNet  MATH  Google Scholar 

  • Cao Y, Zhou Z , Zhang W, Yu Y (2017a) Unsupervised diverse colorization via generative adversarial networks In The European Conference on machine learning and principles and practice of knowledge discovery in databases (ECML-PKDD), LNCS, 10534:151–166

  • Cao Y, Zhou Z, Zhang W, et al (2017b) Unsupervised diverse colorization via generative adversarial networks, Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD) 10534: 151–166

  • Charpiat G, Hofmann M, and Schölkopf B (2008) Automatic image colorization via multimodal predictions 10th European Conference on Computer Vision: part III. Marseille, France, 126–139

  • Chen SY, Zhang JQ, Gao L et al (2022) Active colorization for cartoon line drawings. IEEE Trans Visual Comput Graphics 28(2):1198–1208

    Article  Google Scholar 

  • Chen Y, Luo Y, Ding Y, Yu B (2018) Automatic colorization of images from chinese black and white films based on CNN, 2018 International Conference on Audio, Language and Image Processing (ICALIP), Shanghai, China, Paper ID: 18092463.

  • Cheng Z, Yang Q, Sheng B (2016) Deep Colorization 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, Accession Number: 15801753

  • Chybicki M, Kozakiewicz W, Sielski D et al (2019) Deep cartoon colorizer: an automatic approach for colorization of vintage cartoons. Eng Appl Artif Intell 81:37–46

    Article  Google Scholar 

  • Deng J, Dong W, Socher R, et al (2009) ImageNet: a large-scale hierarchical image database IEEE Computer Vision and Pattern Recognition (CVPR), pp 248–255

  • Deshpande A, Lu J, Yeh M C , et al (2017) Learning diverse image colorization, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA 2877–2885

  • Dong Z, Kamata SI, Breckon TP (2018) Infrared image colorization using S-shape network, 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, 2246

  • Fang F, Wang T, Zeng T, Zhang G (2020) A superpixel-based variational model for image colorization. IEEE Trans Visual Comput Graphics 26(10):2931–2943

    Article  Google Scholar 

  • Faridul HS, Pouli T, Chamaret C et al (2016) Colour mapping: a review of recent methods, extensions and applications. Computer Graphics Forum 35(1):59–88

    Article  Google Scholar 

  • Gupta RK., Chia AY-S, Rajan D, et al (2012) Image colorization using similar images 20th ACM International Conference on Multimedia (ACM MULTIMEDIA), pp 369–378

  • He M, Chen D, Liao J et al (2018) Deep exemplar-based colorization. ACM Transactions Graphics 37(4):1–16

    Google Scholar 

  • He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 1: 770–778

  • Hensman P, Aizawa K (2017) cGAN-based manga colorization using a single training image, 2017 14th IAPR International Conference on Document Analysis and Recognition Kyoto, Japan, 03:72–77

  • Huang YC, Tung YS, Chen JC et al (2005) An adaptive edge detection based colorization algorithm and its applications Proceedings of the 13th annual ACM international conference on Multimedia, Hilton, Singapore, 351–354

  • Iizuka S, Simo-Serra E, Ishikawa H (2016) Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification, ACM SIGGRAPH, Anaheim, CA

  • Irony R, Cohen-Or D, Lischinski D (2005) Colorization by example Sixteenth Eurographics conference on Rendering Techniques, Konstanz, Germany, 201–210

  • Ji G, Wang Z, Zhou L et al (2021) SAR image colorization using multidomain cycle-consistency generative adversarial network. IEEE Geosci Remote Sens Lett 18(2):296–300

    Article  Google Scholar 

  • Ke X, Zou J, Niu Y (2019) End-to-end automatic image annotation based on deep CNN and multi-label data augmentation. IEEE Trans Multimedia 21(8):2093–2106

    Article  Google Scholar 

  • Larsson G, Maire M, Shakhnarovich G (2016a) learning representations for automatic colorization, European Conference on Computer Vision, (ECCV), pp 577–593 (https://tinyclouds.org/colorize/)

  • Larsson G, Maire M, Shakhnarovich G (2016b) Learning representations for automatic colorization European Conference on Computer Vision (ECCV) Amsterdam, Netherlands LNCS, 9908: 577–593

  • Levin A, Lischinski D, Weiss Y (2004) Colorization using optimization. ACM Transactions Graphics 23(3):689–694

    Article  Google Scholar 

  • Li B, Lai Y, John M, Rosin PL (2019) Automatic example-based image colourisation using location-aware cross-scale matching. IEEE Trans Image Process 28(9):4606–4619

    Article  MathSciNet  MATH  Google Scholar 

  • Liang W, Ding D, Wei G (2021) An improved DualGAN for near-infrared image colorization. Infrared Phys Technol 116:103764

    Article  Google Scholar 

  • Liu H, Fu Z, Han J et al (2018) Single satellite imagery simultaneous super-resolution and colorization using multi-task deep neural networks. J vis Commun Image Represent 53:20–30

    Article  Google Scholar 

  • Liu X, Wu W, Li C et al (2022) Reference-guided structure-aware deep sketch colorization for cartoons. Comp Visual Media 8:135–148

    Article  Google Scholar 

  • Luan Q, Wen F, Cohenor D et al (2007) Natural image colorization. EGSR'07. In: Proceedings of the 18th Eurographics conference on Rendering Techniques. 309–320. https://dl.acm.org/doi/10.5555/2383847.2383887

  • Masci J, Meier U, Ciresan D, et al (2011) Stacked convolutional auto-encoders for hierarchical feature extraction, 21st International Conference on Artificial Neural Networks ( ICANN), Espoo, Finland, 14–17

  • Nazeri K, Ng E, Ebrahimi M (2018) Image Colorization Using Generative Adversarial Networks, 2018 International Conference on Articulated Motion and Deformable Objects (AMDO ), LNCS, 10945: 85–94

  • Qayynm U, Ahsan Q, Mahmood Z, Chcmdary MA (2018) Thermal colorization using deep neural network, 2018 15th International Bhurban Conference on Applied Sciences and Technology (IBCAST), Islamabad, Pakistan, Accession Number: 17618209

  • Reinhard E, Ashikhmin M, Gooch B et al (2001) Color transfer between images. IEEE Comput Graphics Appl 21(5):34–41

    Article  Google Scholar 

  • Ren H, Li J, Gao N (2018) Automatic sketch colorization with tandem conditional adversarial networks, 2018 11th International Symposium on Computational Intelligence and Design (ISCID), Hangzhou, China, 01:11-15

  • Shinya A, Mori K, Harada T (2018) Potential improvement of CNN-based colorization for non-natural images, 2018 International Workshop on Advanced Image Technology (IWAIT), Chiang Mai, Thailand, Accession Number: 17806446

  • Shinya A, Mori K, Harada T, Thawonmas R (2018) Potential improvement of CNN-based colorization for non-natural images. 2018 International Workshop on Advanced Image Technology (IWAIT), Chiang Mai, Thailand, 2018, Accession Number: 17806446

  • Song Q, Xu F, Jin YQ (2018) SAR image colorization: converting single-polarization to fully polarimetric using deep neural networks. IEEE Access 6:1647–1661

    Article  Google Scholar 

  • Su YY, Sun HM (2019) Emotion-based color transfer of images using adjustable color combinations. Soft Comput 23(3):1007–1020

    Article  MathSciNet  Google Scholar 

  • Suarez PL, Sappa AD, Vintimilla BX (2018) Near InfraRed imagery colorization, 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, 2018, pp 2237–2241

  • Tang L, Wu D, Wang H et al (2021) An adaptive fuzzy inference approach for color image steganography. Soft Comput 25(16):10987–11004

    Article  Google Scholar 

  • Varga D, Szirányi T (2017) Twin deep convolutional neural network for example-based image colorization, 2017 International Conference on Computer Analysis of Images and Patterns (CAIP), LNCS, 10424: 84–195

  • Viana MS, Junior OM, Contreras RC (2020) An improved local search genetic algorithm with a new mapped adaptive operator applied to pseudo-coloring problem. Symmetry 12(10):1684

    Article  Google Scholar 

  • Welsh T, Ashikhmin M, Mueller K (2002) Transferring color to greyscale images. ACM Transactions on Graphics 21:277–280

    Article  Google Scholar 

  • Xiao Y, Wan L, Leung CS et al (2013) Example-based color transfer for gradient meshes. IEEE Trans Multimedia 15(3):549–560

    Article  Google Scholar 

  • Xiao Y, Jiang A, Liu C et al (2022) Semantic-aware automatic image colorization via unpaired cycle-consistent self-supervised network. Int J Intell Syst 37(2):1222–1238

    Article  Google Scholar 

  • Xiao Y, Zhou P, Zheng Y (2018) Interactive deep colorization with simultaneous global and local inputs, 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, United Kingdom, pp 1887- 1891 (https://github.com/Nanne/conditional-colour)

  • Xin J, Gao C, Jingyu H et al (2018) Multimodal sensor medical image fusion based on nonsubsampled shearlet transform and S-PCNNs in HSV space. Signal Process 153:379–395

    Article  Google Scholar 

  • Xu J, Lu K, Shi X et al (2021) A denseunet generative adversarial network for near-infrared face image colorization. Signal Process 11:108007

    Article  Google Scholar 

  • Zhang R, Zhu JY, Isola P et al (2017) Real-time user-guided image colorization with learned deep priors. ACM Transactions Graphics 36(4):119

    Article  Google Scholar 

  • Zhang R, Isola P, Efros AA (2016) Colorful Image Colorization, 2016 European Conference on Computer Vision (ECCV), LNCS, 9907: 649–66. (http://richzhang.github.io/colorization/)

  • Zhu J, Guo Y, Ma H (2018) A data-driven approach for furniture and indoor scene colorization. IEEE Trans Visual Comput Graphics 24(9):2473–2486

    Article  Google Scholar 

Download references

Acknowledgments

This study is supported by the National Natural Science Foundation of China (Nos. 62101481, 62166047, 62002313, 61862067, 61863036), Key Areas Research Program of Yunnan Province in China (No.202001BB050076), Major Scientific and Technological Project of Yunnan Province (No. 202202AD080002), Yunnan Fundamental Research Projects of China (Nos. 202201AU070033, 202201AT070112), Science and Technology Innovation Team Project of Yunnan Province (No. 2017HC012), the Fund Project of Yunnan Province Education Department (No.2022j0008), Key Laboratory in Software Engineering of Yunnan Province (No. 2020SE408), and the open project of Engineering Research Center of Cyberspace in 2021-2022 (No. KJAQ202112012).

Funding

The authors have not disclosed any funding.

Author information

Authors and Affiliations

  1. School of Software, Yunnan University, Kunming, 650091, Yunnan, China

    Xin Jin, Yide Di, Qian Jiang, Xing Chu, Qing Duan, Shaowen Yao & Wei Zhou

Authors
  1. Xin Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yide Di
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qian Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xing Chu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qing Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shaowen Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Wei Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Zhou.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest to publish the paper.

Human and animal rights

This article does not contain any studies with human or animal subjects performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jin, X., Di, Y., Jiang, Q. et al. Image colorization using deep convolutional auto-encoder with multi-skip connections. Soft Comput 27, 3037–3052 (2023). https://doi.org/10.1007/s00500-022-07483-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-022-07483-0

Keywords

Search

Navigation