Skip to main content
SpringerLink
Log in

Scene guided colorization using neural networks

  • Deep Learning for Biomedical and Healthcare Applications
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This paper presents a novel, fully automatic method to grayscale images colorization using a scene guided neural network. In our proposed method, given a training set of both grayscale images and their corresponding color images, we first extract features of each grayscale pixel. These features, together with the corresponding RGB values of that pixel are input to train a colorization neural network for each given scene. To improve the performance of colorization, in both speed and results, we further classify the input and training images into different scene classes. We adopt a linear image classification method to generate a scene guided codebook and use it to determine the scene class of the input image. The preliminary colorization result is then generated by the corresponding trained neural network of the scene class of the input image. Finally, an image guided filter is used to refine colorized images. Inspired by the recent success in deep learning techniques which provide stabilizing modeling of large-scale medical image data, the proposed paper formulating the enhancement and colorization problem, so that colorization techniques can be directly used to ensure medical images with high quality. The experimental results on a broad range of images demonstrate that our method has better colorization performance as compared to that of the state-of-the-art algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. MacIntyre J (2013) Applications of neural computing in the twenty-first century and 21 years of Neural Computing & Applications. Neural Comput Appl 23(3–4):657–665

    Article  Google Scholar 

  2. Wan S, Liang Y, Zhang Y, Guizani M (2018) Deep multi-layer perceptron classifier for behavior analysis to estimate parkinsons disease severity using smartphones. IEEE Access 6(99):36825–36833

    Article  Google Scholar 

  3. Li D, Zhang C, Wen J (2015) A note on compact finite difference method for reaction-diffusion equations with delay. Appl Math Model 39:1749–1754

    Article  MathSciNet  Google Scholar 

  4. Li D, Zhang J (2016) Efficient implementation to numerically solve the nonlinear time fractional parabolic problems on unbounded spatial domain. J Comput Phys 322:415–428

    Article  MathSciNet  Google Scholar 

  5. Huang YC, Tung YS, Chen JC, Wang SW, Wu JL (2005) An adaptive edge detection based colorization algorithm and its applications. In: ACM international conference on multimedia, ACM, pp 351–354

  6. Levin A, Lischinski D, Weiss Y (2004) Colorization using optimization. In: ACM transactions on graphics (TOG), ACM, vol. 23, pp 689–694

  7. Luan Q, Wen F, Cohen D, Liang L, Xu YQ, Shum HY (2007) Natural image colorization. In: Eurographics conference on rendering techniques, Eurographics Association, pp 309–320

  8. Qu Y, Wong TT, Heng PA (2006) Manga colorization. In: ACM transactions on graphics (TOG), ACM, vol 25, pp 1214–1220

  9. Yatziv L, Sapiro G (2006) Fast image and video colorization using chrominance blending. IEEE Trans Image Process 15(5):1120–1129

    Article  Google Scholar 

  10. Charpiat G, Hofmann M, Scholkopf B (2008) Automatic image colorization via multimodal predictions. In: European conference on computer vision, Springer, pp 126–139

  11. Chia AYS, Zhuo SJ, Gupta RK, Tai YW, Cho SY, Tan P, Lin S (2011) Semantic colorization wit internet images. ACM Trans Graph (TOG) 30(6):1–8

    Article  Google Scholar 

  12. Gupta RK, Chia AYS, Rajan D, Ng ES.n, Huang ZY (2012) Image colorization using similar images. In: ACM international conference on multimedia, pp 369–378

  13. Irony R, Cohen-Or D, Lischinski D (2005) Colorization by example. In: Eurographics Symposium on rendering, vol 2, Citeseer

  14. Liu XP, Wan L, Qu Y, Wong TT, Lin S, Leung CS, Heng PA (2008) Intrinsic colorization. ACM Trans Graph (TOG) 27(5):152

    Article  Google Scholar 

  15. Welsh T, Ashikhmin M, Mueller K (2002) Transferring color to greyscale images. In: ACM transactions on graphics (TOG), ACM, vol 21, pp 277–280

  16. Hertzmann A, Jacobs CE, Oliver N, Curless B, Salesin DH (2001) Image analogies. In: Computer graphics and interactive techniques, ACM, pp 327–340

  17. Reinhard E, Adhikhmin M, Gooch B, Shirley P (2001) Color transfer between images. IEEE Comput Graph Appl 21(5):34–41

    Article  Google Scholar 

  18. Cheng ZZ, Yang QX, Sheng B (2015) Deep colorization. In: IEEE international conference on computer vision, pp 415–423

  19. Deshpande A, Rock J, Forsyth D (2015) Learning large-scale automatic image colorization. In: IEEE international conference on computer vision, pp 567–575

  20. Iizuka S, Serra ES, Ishikawa H (2016) Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans Graph (TOG) 35(4):110

    Article  Google Scholar 

  21. Zhang R, Isola P, Efros AA (2016) Colorful image colorization. In: European conference on computer vision, pp 649–666

  22. Liu GH, Yang JY (2013) Content-based image retrieval using color difference histogram. Pattern Recognit 46(1):188–198

    Article  Google Scholar 

  23. Yang J, Liu C, Zhang L (2010) Color space normalization: enhancing the discriminating power of color spaces for face recognition. Pattern Recognit 43(4):1454–1466

    Article  Google Scholar 

  24. Patterson G, Xu C, Su H, Hays J (2014) The sun attribute database: beyond categories for deeper scene understanding. Int J Comput Vis 108(1–2):59–81

    Article  Google Scholar 

  25. Yang JC, Yu K, Gong YH, Huang T (2009) Linear spatial pyramid matching using sparse coding for image classification. In: Computer vision and pattern recognition. IEEE, pp 1794–1801

  26. He KM, Sun J, Tang X (2013) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35(6):1397–1409

    Article  Google Scholar 

  27. Haghighat M, Zonouz S, Mohamed AM (2015) Cloudid: trustworthy cloud-based and cross-enterprise biometric identification. Expert Syst Appl 42(21):7905–7916

    Article  Google Scholar 

  28. Tola E, Lepetit V, Fua P (2008) A fast local descriptor for dense matching. In: IEEE conference on computer vision and pattern recognition. IEEE, pp 1–8

  29. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: International conference on neural information processing systems, pp 1097–1105

  30. Petschnigg G, Szeliski R, Agrawala M, Cohen M, Hoppe H, Toyama K (2004) Digital photography with flash and no-flash image pairs. ACM Trans Graph (TOG) 23(3):664–672

    Article  Google Scholar 

  31. Hore A, Ziou D (2010) Image quality metrics: Psnr vs. ssim. International conference on pattern recognition (ICPR). IEEE, pp 2366–2369

  32. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: IEEE conference on computer vision and pattern recognition, pp 3431–3440

  33. Liu D, Jiang Y, Pei M (2018) Emotional image color transfer via deep learning. Pattern Recognit Lett 110:16–22

    Article  Google Scholar 

  34. Su Z, Liang X, Guo J (2018) An edge-refined vectorized deep colorization model for grayscale-to-color images. Neurocomputing 311:305–315

    Article  Google Scholar 

  35. Daugman JG (1988) Complete discrete 2-D Gabor transforms by neural networks for image analysis and compression. IEEE Trans Acoust Speech Signal Process 36(7):1169–1179

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their valuable comments which have helped to improve the quality of this paper. This work was partially supported by the Chinese Scholarship Council (grant no. 201506290117), the Natural Sciences and Engineering Research Council of Canada (NSERC) and the University of Alberta.

Author information

Authors and Affiliations

  1. School of Automation, Northwestern Polytechnical University, Xi’an, 710129, China

    Yu Xia & Shiru Qu

  2. School of Information and Safety Engineering, Zhongnan University of Economics and Law, Wuhan, 430073, China

    Shaohua Wan

Authors
  1. Yu Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shiru Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shaohua Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shaohua Wan.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interests regarding the publication of this article.

Appendix A. Supplementary material

Appendix A. Supplementary material

Supplementary data associated with this article can be found in Fig. 9 that presents more colorization images obtained from our method with respect to the ground truth color images. The experimental results on a broad range of images demonstrate that there are almost not visible artifacts in the images colorized using the proposed method, and these results are visually very similar to the ground truth.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xia, Y., Qu, S. & Wan, S. Scene guided colorization using neural networks. Neural Comput & Applic 34, 11083–11096 (2022). https://doi.org/10.1007/s00521-018-3828-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-018-3828-z

Keywords

Search

Navigation