Skip to main content

Advertisement

Springer Nature Link
Log in

Color vision deficiency datasets & recoloring evaluation using GANs

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

People with Color Vision Deficiency (CVD) cannot distinguish some color combinations under normal situations. Recoloring becomes a necessary adaptation procedure. In this paper, in order to adaptively find the key color components in an image, we first propose a self-adapting recoloring method with an Improved Octree Quantification Method (IOQM). Second, we design a screening tool of CVD datasets that is used to integrate multiple recoloring methods. Third, a CVD dataset is constructed with the help of our designed screening tool. Our dataset consists of 2313 pairs of training images and 771 pairs of testing images. Fourth, multiple GANs i.e., pix2pix-GAN [1], Cycle-GAN [2], Bicycle-GAN [3] are used for colorblind data conversion. This is the first ever effort in this research area using GANs. Experimental results show that pix2pix-GAN [1] can effectively recolor unrecognizable colors for people with CVD, and we predict that this dataset can provide some help for color blind images recoloring. Datasets and source are available at: https://github.com/doubletry/pix2pix, https://github.com/doubletry/CycleGAN and https://github.com/doubletry/BicycleGAN.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. website1: http://labs.tineye.com/multicolr/

  2. website2:http://www.color-blindness.com/coblis-color-blindness-simulator/

References

  1. Baker S, Scharstein D, Lewis J, Roth S, Black MJ, Szeliski R (2011) A database and evaluation methodology for optical flow. Int J Comput Vis 92(1):1–31

    Article  Google Scholar 

  2. Bansal A, Russell B, Gupta A (2016) Marr revisited: 2d-3d alignment via surface normal prediction. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5965–5974

  3. Brettel H, Viénot F, Mollon JD (1997) Computerized simulation of color appearance for dichromats. JOSA A 14(10):2647–2655

    Article  Google Scholar 

  4. Cao Z, Simon T, Wei S-E, Sheikh Y (2016) Realtime multi-person 2d pose estimation using part affinity fields, arXiv:1611.08050

  5. Chen L-C (2018a) Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Patt Anal Mach Intell 40(4):834–848

    Article  Google Scholar 

  6. Chen X, Duan Y, Houthooft R, Schulman J, Sutskever I, Abbeel P (2016) Infogan: Interpretable representation learning by information maximizing generative adversarial nets, NIPS

  7. Chen L.-C., Papandreou G, Kokkinos I, Murphy K, Yuille AL (2014) Semantic image segmentation with deep convolutional nets and fully connected crfs, arXiv:1412.7062

  8. Chen L.-C., Papandreou G, Schroff F, Adam H (2017) Rethinking atrous convolution for semantic image segmentation, arXiv:1706.05587

  9. Chen L.-C., Zhu Y, Papandreou G, Schroff F, Adam H (2018b) Encoder-decoder with atrous separable convolution for semantic image segmentation, arXiv:1802.02611

  10. Doliotis P, Tsekouras G, Anagnostopoulos C-N, Athitsos V (2009) Intelligent modification of colors in digitized paintings for enhancing the visual perception of color-blind viewers. In: IFIP International conference on artificial intelligence applications and innovations. Springer, New York, pp 293–301

  11. Dollár P, Zitnick CL (2013) Structured forests for fast edge detection. In: Proceedings of the IEEE international conference on computer vision, pp 1841–1848

  12. Donahue J, krähenbühl P, Darrell T (2016) Adversarial feature learning, arXiv:1605.09782

  13. Dosovitskiy A, Fischer P, Ilg E, Hausser P, Hazirbas C, Golkov V, Van Der Smagt P, Cremers D, Brox T (2015) Flownet: Learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 2758–2766

  14. Dumoulin V, Belghazi I, Poole B, Mastropietro O, Lamb A, Arjovsky M, Courville A (2016) Adversarially learned inference, ICLR

  15. Eigen D, Fergus R (2015) Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of the IEEE international conference on computer vision, pp 2650–2658

  16. Eigen D, Puhrsch C, Fergus R (2014) Depth map prediction from a single image using a multi-scale deep network. In: Advances in neural information processing systems, pp 2366–2374

  17. Everingham M, Eslami SMA, Gool LJV, Williams CKI, Winn JM, Zisserman A (2015) The pascal visual object classes challenge: a retrospective,. Int J Comput Vision 111(1):98–136. [Online]. Available: https://doi.org/10.1007/s11263-014-0733-5

    Article  Google Scholar 

  18. Fan D-P, Cheng M-M, Liu J-J, Gao S-H, Hou Q, Borji A (2018) Salient objects in clutter: bringing salient object detection to the foreground. In: Proceedings of the european conference on computer vision (ECCV), pp 186–202

  19. Fan D-P, Wang W, Cheng M-M, Shen J (2019) Shifting more attention to video salient object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, conference proceedings, pp 8554–8564

  20. Fergus R, Fergus R, Fergus R, Fergus R (2015) Deep generative image models using a laplacian pyramid of adversarial networks. In: International conference on neural information processing systems, pp 1486–1494

  21. Fluck D (2006) Coblis - color blindness simulator. [Online]. Available: http://www.color-blindness.com/coblis-color-blindness-simulator/

  22. Fu K, Zhao Q, Gu IY-H (2018) Refinet: a deep segmentation assisted refinement network for salient object detection. IEEE Trans Multimed 21(2):457–469

    Article  Google Scholar 

  23. Fu K, Zhao Q, Gu IY-H, Yang J (2019) Deepside: a general deep framework for salient object detection. Neurocomputing 356:69–82

    Article  Google Scholar 

  24. Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? the kitti vision benchmark suite. In: Conference on computer vision and pattern recognition (CVPR)

  25. Gervautz M, Purgathofer W (1988) A simple method for color quantization: octree quantization. In: New trends in computer graphics. Springer, New York, pp 219–231

  26. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587

  27. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: International conference on neural information processing systems, pp 2672–2680

  28. He K, Gkioxari G, Dollár P., Girshick R (2017) Mask r-cnn. In: Computer vision (ICCV), 2017 IEEE international conference on. IEEE, pp 2980–2988

  29. Huang JB, Chen CS, Jen TC, Wang SJ (2009) Image recolorization for the colorblind. In: IEEE International conference on acoustics, speech and signal processing, pp 1161–1164

  30. Huang CR, Chiu KC, Chen CS (2010) Key color priority based image recoloring for dichromats. Adv Multimed Inform Process - PCM 2010 6298:637–647

    Article  Google Scholar 

  31. Huang L, Yang Y, Deng Y, Yu Y (2015) Densebox: Unifying landmark localization with end to end object detection, arXiv:1509.04874

  32. Isola P, Zhu J-Y, Zhou T, Efros AA (2016) Image-to-image translation with conditional adversarial networks. CVPR 5967–5976

  33. Jégou S., Drozdzal M, Vazquez D, Romero A, Bengio Y (2017) The one hundred layers tiramisu: fully convolutional densenets for semantic segmentation. In: Computer vision and pattern recognition workshops (CVPRW), 2017 IEEE Conference on. IEEE, pp 1175–1183

  34. Jeong J-Y, Kim H-J, Wang T-S, Yoon Y-J, Ko S-J (2011) An efficient re-coloring method with information preserving for the color-blind. IEEE Transa Consumer Electron 57(4)

  35. Katsuhiro N, Manami T, Hiroshi S, Hiroshi O, Mu S, Atsushi H, Isao M, Shin’Ichi I, Nobuyuki F, Kazunori K (2016) A way of color image processing for the colorblinds. Bull Hiroshima Mercant Marine College 38

  36. Khurge DS, Peshwani B (2015) Modifying image appearance to improve information content for color blind viewers. In: Computing Communication Control and Automation (ICCUBEA), 2015 International Conference on. IEEE, pp 611–614

  37. Kim H-J, Jeong J-Y, Yoon Y-J, Kim Y-H, Ko S-J (2012) Color modification for color-blind viewers using the dynamic color transformation. In: Consumer electronics (ICCE), 2012, IEEE international conference on. IEEE, pp 602–603

  38. Kim YK, Kim KW, Yang X (2007) Real time traffic light recognition system for color vision deficiencies. In: Mechatronics and automation, 2007. ICMA 2007. International conference on IEEE, pp 76–81

  39. Kingma DP, Welling M (2013) Auto-encoding variational bayes, ICLR

  40. Larsen ABL, Larochelle H, Winther O (2015) Autoencoding beyond pixels using a learned similarity metric. ICML 1558–1566

  41. Lu X, Wang W, Ma C, Shen J, Shao L, Porikli F (2019) See more, know more: unsupervised video object segmentation with co-attention siamese networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, conference proceedings, pp 3623–3632

  42. Martin DR, Fowlkes CC, Malik J (2004) Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Trans Patt Anal Mach Intell 26(5):530–549

    Article  Google Scholar 

  43. Martin CE, Keller J, Rogers SK, Kabrinsky M (2000) Color blindness and a color human visual system model. IEEE Trans Syst Man Cyber - Part A: Syst Humans 30(4):494–500

    Article  Google Scholar 

  44. Mathieu M, Zhao J, Sprechmann P, Ramesh A, Lecun Y (2016) Disentangling factors of variation in deep representations using adversarial training. NIPS 5040–5048

  45. Maurer CR, Qi R, Raghavan V (2003) A linear time algorithm for computing exact euclidean distance transforms of binary images in arbitrary dimensions. IEEE Trans Patt Anal Mach Intell 25(2):265–270

    Article  Google Scholar 

  46. Milić N., Belhadj F, Dragoljub N (2015) The customized daltonization method using discernible colour bins. In: Colour and visual computing symposium (CVCS), 2015. IEEE, pp 1–6

  47. Milić N, Hoffmann M, Tómács T, Novaković D, Milosavljević B (2015) A content-dependent naturalness-preserving daltonization method for dichromatic and anomalous trichromatic color vision deficiencies. J Imaging Sci Technol 59(1):10 504–1

    Article  Google Scholar 

  48. Orii H, Kawano H, Maeda H, Kouda T (2014) Color conversion algorithm for color blindness using self-organizing map. In: Soft computing and intelligent systems (SCIS), 2014 joint 7th international conference on and advanced intelligent systems (ISIS), 15th international symposium on. IEEE, pp 910–913

  49. Rasche K, Geist R, Westall J (2005) Detail preserving reproduction of color images for monochromats and dichromats. IEEE Comput Graph Appl 25 (3):22–30

    Article  Google Scholar 

  50. Russell BC, Torralba A, Murphy KP, Freeman WT (2008) Labelme: a database and web-based tool for image annotation. Int J Comput Vision 77(1-3):157–173

    Article  Google Scholar 

  51. Sharpe LT, Stockman A, Jägle H, Nathans J (1999) Opsin genes, cone photopigments, color vision, and color blindness. Color vision: From genes to perception, 351

  52. Shelhamer E, long J, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 3431–3440

  53. Silberman N, Hoiem D, Kohli P, Fergus R (2012) Indoor segmentation and support inference from rgbd images. In: European conference on computer vision. Springer, New York, pp 746–760

  54. Stenetorp P, Pyysalo S, Topić G, Ohta T, Ananiadou S, Tsujii J (2012) Brat : a web-based tool for NLP-assisted text annotation. In: Proceedings of the demonstrations session at EACL 2012. Avignon France: Association for computational linguistics

  55. Teney D, Hebert M (2016) Learning to extract motion from videos in convolutional neural networks. In: Asian conference on computer vision. Springer, New York, pp 412–428

  56. Tzutalin (2015) Labelimg. Git code. [Online]. Available: https://github.com/tzutalin/labelImg

  57. Vondrick C, Patterson D, Ramanan D (2013) Efficiently scaling up crowdsourced video annotation. Int J Comput Vis 101(1):184–204

    Article  Google Scholar 

  58. Wandell BA (1995) Foundations of vision. Sinauer Associates Sunderland, MA, vol 8

  59. Wang X, Fouhey D, Gupta A (2015) Designing deep networks for surface normal estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 539–547

  60. Wei S-E, Ramakrishna V, Kanade T, Sheikh Y (2016) Convolutional pose machines. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4724–4732

  61. Xie S, Tu Z (2015) Holistically-nested edge detection. In: Proceedings of the IEEE international conference on computer vision, pp 1395–1403

  62. Zhao J-X, Liu J-J, Fan D-P, Cao Y, Yang J, Cheng M-M (2019) Egnet: Edge guidance network for salient object detection. In: Proceedings of the IEEE international conference on computer vision, conference proceedings, pp 8779–8788

  63. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2881–2890

  64. Zhou B, Zhao H, Puig X, Fidler S, Barriuso A, Torralba A (2017) Scene parsing through ade20k dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  65. Zhu J-Y, Krähenbühl P, Shechtman E, Efros AA (2016) Generative visual manipulation on the natural image manifold. In: European conference on computer vision. Springer, New York, pp 597–613

  66. Zhu J-Y, Park T, Isola P (2017) Unpaired image-to-image translation using cycle-consistent adversarial networkss. In: Computer vision (ICCV), 2017 IEEE international conference on, pp 2242–2251

  67. Zhu J-Y, Zhang R, Pathak D, Darrell T, Efros AA, Wang O, Shechtman (2017) Toward multimodal image-to-image translation. In: Advances in neural information processing systems

Download references

Acknowledgements

Supported by National Key R&D Program of China under Grant No. 2019YFB1311600 & Ningbo 2025 Key Project of Science and Technology Innovation (2018B10071).

Author information

Authors and Affiliations

  1. Xidian University, Xi’an, China

    Hongsheng Li, Liang Zhang, Xiangdong Zhang, Meili Zhang, Guangming Zhu & Peiyi Shen

  2. Shanghai National Engineering Research Center for Broadband Networks & Applications, Shanghai, China

    Ping Li

  3. The University of Western Australia, Perth, Australia

    Mohammed Bennamoun & Syed Afaq Ali Shah

Authors
  1. Hongsheng Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Liang Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Xiangdong Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Meili Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Guangming Zhu
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Peiyi Shen
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Ping Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Mohammed Bennamoun
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. Syed Afaq Ali Shah
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Liang Zhang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A

Appendix A

More recoloring results are shown in this appendix.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Zhang, L., Zhang, X. et al. Color vision deficiency datasets & recoloring evaluation using GANs. Multimed Tools Appl 79, 27583–27614 (2020). https://doi.org/10.1007/s11042-020-09299-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09299-2

Keywords