Keywords

1 Introduction

With the explosive growth of multimedia technology, perceiving color becomes more and more important for effective visual communication. Normal vision is based on the response to photons in L-, M- and S- cones located in the retina of the eyes. However, color vision deficiency (CVD) presents when there exists a deficiency in one of the cones, and it can be divided into protanopia, deuteranopia and tritanopia according to the deficient cone. In this paper, we focus on protanopia and deuteranopia since they occupy 98% of CVD, who cannot perceive red and green under normal situation [1]. We show an example of how human with CVD perceives colors in Fig. 1. The significant information disappears in Fig. 1(b) and (c), which are perceived by protanopia and deuteranopia, respectively. It has been demonstrated that CVD causes people considerable inconvenience, because the ability to distinguish colors is useful in many daily activities such as recognizing road signs and traffic lights [2]. Therefore, it is important to enhance color perceptibility for the human with CVD.

Fig. 1.
figure 1

An example of how human with CVD perceives colors. (a) The image that perceived by human with normal vision. (b) The image that perceived by human with protanopia. (c) The image that perceived by human with deuteranopia.

Full size image

Various recoloring methods have been proposed to enhance color perceptibility for human with CVD. And these methods can be categorized into rule-based and optimization-based methods.

In the optimization-based methods [3,4,5,6,7], recoloring is performed by minimizing an objective function based on some metrics. Huang [3] and Huang [4] obtained optimal colors by minimizing the difference between original and recolored image in Lab color space. And Milić [5, 6] found the optimal choice of discernible color bin for each color in the original image. These methods are easy to fall into a shallow local minimum of the objective function and cannot meet the real-time requirement because of exhaustive search for minimum.

The rule-based methods [8,9,10,11,12] need less time than optimization-based methods since they only require linear operation. There are two factors influencing the performance. One is to judge unrecognizable colors for human with CVD and the other is to recolor these unrecognizable colors using a mapping matrix. Doliotis [8] and Khurge [9] found unrecognizable colors by comparing the difference between original and simulated image with a pre-defined threshold. And Kim [10, 11] found unrecognizable colors through a pre-defined red pixel mask for protanopia. These methods involve some pre-defined parameters, which has great impact on the performance. When recoloring unrecognizable colors, Khurge [9] proposed an experience-based matrix, but this matrix cannot guarantee all the unrecognizable colors are recolored accurately. Doliotis [8] and Kim [10] recolored the unrecognizable colors by multiplying a given matrix iteratively until the stopping iteration condition was satisfied. The stopping iteration condition is also a pre-defined parameter similar as unrecognizable colors judgment, which cannot guarantee high contrast between different colors in recolored images. And these methods recolored all the unrecognizable colors simultaneously, which need more iterations to make sure all the recolored colors are recognizable for human with CVD.

In this paper, an efficient rule-based recoloring method is proposed for human with CVD. First, key color confidence is introduced to judge and recolor one unrecognizable color at a time so that recoloring procedure can be done sequentially instead of simultaneously, which can speed up the execution time greatly. Second, color differences between one color and other colors with higher confidence are taken as the unrecognizable benchmark for this color, which cannot involve any pre-defined parameters. Third, an optimal recolored color for this unrecognizable color is obtained when these differences have been maintained or enhanced after recoloring, which can guarantee high contrast between different colors in the recolored image.

The rest of this paper is organized as follows. Section 2 describes our proposed method in detail. Section 3 demonstrates the effectiveness of our proposed method in terms of subjective quality, objective quality and execution time. And Sect. 4 concludes this paper.

2 Proposed Algorithm

Figure 2 depicts a flowchart of our proposed recoloring method:

  1. 1.

    Key colors are extracted in RGB color space. RGB color space is a convenient color model for image processing because human visual system works in a similar way to it [13]. Since people cannot distinguish colors with small color difference under normal situation, key colors \(C^{0}=\{C_{1}^{0},C_{2}^{0},...,C_{N}^{0}\}\) are extracted from an original image using the K-means [14], in which each key color \(C_{i}^{0}\) represents a color of the cluster center. And then the key colors perceived by CVD \(C^{d}=\{C_{1}^{d},C_{2}^{d},...,C_{N}^{d}\}\) are obtained using a well-known color blind simulation algorithm proposed by Brettel [15, 16];

  2. 2.

    The key color confidences are obtained according to the differences between \(C^{0}\) and \(C^{d}\), and all key color in \(C^{0}\) are ordered according to their corresponding confidence, which is illustrated in Sect. 2.1 in detail;

  3. 3.

    From the highest to lowest confidence, judge whether one key color \(C_{i}^{0}\) is unrecognizable for human with CVD by taking the differences between \(C_{i}^{0}\) and other colors with higher confidence \(C_{j}^{0}(j<i)\) as judgment benchmark. If it is unrecognizable, go to step 4, otherwise go to step 5, which is illustrated in Sect. 2.2 in detail;

  4. 4.

    A unrecognizable color \(C_{i}^{0}\) will be recolored to the recolored color \(O_{i}^{0}\) by multiplying with a mapping matrix iteratively, until \(C_{i}^{0}\) is not confused with those recolored colors with higher confidence \(O_{j}^{0}(j<i)\), which is also illustrated in Sect. 2.2 in detail;

  5. 5.

    Judge whether all key colors in \(C^{0}\) have been recolored. If it is yes, then perform step 6, otherwise go back to step 3;

  6. 6.

    All pixels in original image are recolored through interpolation. We tend to maintain the perceptibility distance between a pixel and its corresponding key color after recoloring. Given a pixel \(P_{j}\) in original image and its corresponding key color \(C_{i}^{0}\), the recolored color \(R_{j}\) for pixel \(P_{j}\) is obtained using Eq. (1).

    $$\begin{aligned} R_{j}=O_{i}^{0}+(P_{j}-C_{i}^{0}) \end{aligned}$$
    (1)
Fig. 2.
figure 2

The flowchart of our proposed algorithm.

Full size image

2.1 Key Color Confidence Calculation

In the existing rule-based recoloring methods [8,9,10,11,12], all unrecognizable colors for human with CVD are judged and recolored simultaneously, which needs more iterations to make sure all unrecognizable colors are recolored accurately. To speed up execution time, we introduce key color confidence to judge and recolor one color at a time so that recoloring procedure can be done sequentially. And an optimal recolored color can be obtained as long as the recolored color is not confused with other recolored colors with higher confidence.

Key color confidence is the confidence whether one color is recognizable for human with CVD. A key color with higher confidence should be taken as the benchmark for recoloring other colors with lower confidence. To obtain the key color confidences, we first calculate the color differences \(D=\{D_{1},D_{2},...,D_{N}\}\) between \(C^{0}\) and \(C^{d}\). Then, one key color \(C_{i}^{0}\) with smaller color difference \(D_{i}\) is assigned a higher confidence. Finally, all key colors in \(C^{0}\) are ordered according to their corresponding confidence. We define color differences between \(C^{0}\) and \(C^{d}\) using a well-known weighted Euclidean distance, as shown in Eq. (2), which approximates to the perceptibility distance of human [17].

$$\begin{aligned} D_{i}=\sqrt{3*{\varDelta R_{i}}^{2}+4*{\varDelta G_{i}}^{2}+2*{\varDelta B_{i}}^{2}} \end{aligned}$$
(2)

where \(\varDelta R_{i}=C_{i,R}^{0}-C_{i,R}^{d}\), \(\varDelta G_{i}=C_{i,G}^{0}-C_{i,G}^{d}\) and \(\varDelta B_{i}=C_{i,B}^{0}-C_{i,B}^{d}\) are the differences between three components of \(C^{0}\) and \(C^{d}\), respectively.

To speed up the execution time, we then judge and recolor one unrecognizable color for human with CVD at a time from the highest to lowest confidence, so that recoloring process can be done sequentially instead of simultaneously.

2.2 Unrecognizable Colors Judgment and Recoloring

In the existing rule-based recoloring methods [8,9,10,11,12], the unrecognizable colors judgement and the stopping iteration condition involve some pre-defined parameters. The performance is sensitive to these parameters and the contrast between different colors in recolored image is low. To reduce the influence of the pre-defined parameters, we propose to take the differences between one color and other colors with higher confidence as unrecognizable benchmark for this color. And we obtain an optimal recolored color for this unrecognizable color when these differences have been maintained or enhanced in the iterative recoloring, which can maximize the contrast between different colors in the recolored image.

The differences between one key color \(C_{i}^{0}\) and other colors with higher confidence \(C_{j}^{0}(j<i)\) are the benchmark for judging whether \(C_{i}^{0}\) is unrecognizable for human with CVD. If there exists one color with higher confidence \(O_{j}^{d}(j<i)\) in \(O^{d}\) (recolored colors perceived by CVD), and when the difference between \(O_{j}^{d}\) and \(C_{i}^{0}\) is smaller than the corresponding difference between \(C_{j}^{0}\) and \(C_{i}^{0}\), we regard \(C_{i}^{0}\) as an unrecognizable color.

Then the unrecognizable color \(C_{i}^{0}\) is recolored in LMS color space that specifies colors in terms of the relative excitations of L-, M- and S- cones. First, \([L_{i}^{0},M_{i}^{0},S_{i}^{0}]\) and \([L_{i}^{d},M_{i}^{d},S_{i}^{d}]\) are obtained by transforming \(C_{i}^{0}\) and \(C_{i}^{d}\) from RGB to LMS color space. Then, the recolored color \([L_{i}^{'},M_{i}^{'},S_{i}^{'}]\) in LMS color space is obtained according to Eq. (3).

$$\begin{aligned}{}[L_{i}^{'},M_{i}^{'},S_{i}^{'}]=[L_{i}^{0},M_{i}^{0},S_{i}^{0}]+\alpha *T*([L_{i}^{0},M_{i}^{0},S_{i}^{0}]-[L_{i}^{d},M_{i}^{d},S_{i}^{d}]) \end{aligned}$$
(3)

where T is a mapping matrix proposed in [16], as shown in Eq. (4). And \( \alpha \) is a directional parameter that controls the direction of recoloring. If the recolored color \(O_{i}^{0}\) has exceeded the range of RGB color space before obtaining an optimal recolored color, we move to the opposite direction of recoloring according to Eq. (5).

$$\begin{aligned} T = { \left[ \begin{array}{ccc} 0 &{} 2.02344 &{} -2.52581 \\ 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} 1 \end{array} \right] } \end{aligned}$$
(4)
$$\begin{aligned} \alpha = -0.75*\alpha \end{aligned}$$
(5)

Finally, the recolored color \([L_{i}^{'},M_{i}^{'},S_{i}^{'}]\) is transformed back to RGB color space to obtain the recolored key color \(O_{i}^{0}\), and then \(O_{i}^{0}\) is simulated to obtain the corresponding recolored color perceived by CVD \(O_{i}^{d}\). If the difference between \(O_{i}^{d}\) and one of the colors with higher confidence \(O_{j}^{d}(j<i)\) in \(O^{d}\) is lower than the corresponding difference between \(C_{i}^{0}\) and \(C_{j}^{0}\), we still regard \(O_{i}^{0}\) as an unsatisfactory recolored color for \(C_{i}^{0}\). Then we continue to perform recoloring until the differences between \(O_{i}^{d}\) and all recolored colors with high confidence perceived by CVD \(O_{j}^{d}(j<i)\) are higher than the corresponding differences between \(C_{i}^{0}\) and \(C_{j}^{0}\).

3 Experimental Results and Analysis

Our proposed method is compared with algorithms including Doliotis [8], Huang [4] and Khurge [9] in terms of subjective quality, objective quality and execution speed. The test images consist of both natural images and Ishihara test images. We compile all the algorithms using Matlab 2013a on an Intel(R) Core(TM) i3-2310M CPU @ 2.10 GHz. In our proposed method, all images are clustered into 6 clusters in step 1 and the directional parameter is initialized to be 1 in step 4.

3.1 Subjective Quality Comparison

A recoloring algorithm can be considered to have a good performance if the resulting recolored images can preserve the overall characteristics of the original images and achieve a high contrast between different colors for human with CVD. First, all the algorithms are tested with two natural images and the experiment results are shown in Figs. 3, 4 and 5.

Fig. 3.
figure 3

(a) The original nature image. (b) The recolored image using our proposed method. (c) The recolored image using Doliotis’s method [8]. (d) The recolored image using Huang’s method [4]. (e) The recolored image using Khurge’s method [9]. (f)–(j) The corresponding images perceived by protanopia. (Color figure online)

Full size image

As shown in Fig. 3(a) and (f), the green leaves and red fruits are unrecognizable for human with protanopia. Doliotis [8] recolors unrecognizable colors iteratively until the stopping iteration condition is satisfied. As shown in Fig. 3(h), these two colors are still confused by human with protanopia, since the stopping iteration condition is not appropriate. Huang [4] proposes a optimization-based method. As shown in Fig. 3(i), the contrast between these two colors is still low for human with protanopia, since this method is likely to fall into the shallow local minimum. And the contrast between these two colors in Fig. 3(j) is also low for human with protanopia, since Khurge [9] recolors unrecognizable color according to an experience-based matrix.

As shown in Fig. 4(a) and (f), red is the only one unrecognizable color for human with protanopia. In Doliotis’s [8] method, red have been recolored to light green, but it is confused with gray, as shown in Fig. 4(h). In Huang’s [4] method, all colors in Fig. 4(a) are recolored regardless of whether they are recognizable for CVD. And the contrast between different colors becomes very low, as shown in Fig. 4(i). Similar to Doliotis, Khurge [9] recolors red to light pink, but it is confused with gray, as shown in Fig. 4(j).

Fig. 4.
figure 4

(a) The original natural image. (b) The recolored image using our proposed method. (c) The recolored image using Doliotis’s method [8]. (d) The recolored image using Huang’s method [4]. (e) The recolored image using Khurge’s method [9]. (f)–(j) The corresponding images perceived by protanopia. (Color figure online)

Full size image

As shown in Fig. 5(a) and (f), red flower is confused with the green background for human with protanopia. In Doliotis’s [8] method, red have been recolored to light green, but it is still easily confused with background, as shown in Fig. 5(h). In Huang’s [4] method, red flower is recolored to dark green, but the contrast between flower and background is still lower than original image, as shown in Fig. 5(i). Similar to Doliotis, Khurge [9] recolors red flower and background to similar grey, so the contrast between flower and background is very low, as shown in Fig. 5(j).

Fig. 5.
figure 5

(a) The original Ishihara test image. (b) The recolored image using our proposed method. (c) The recolored image using Doliotis’s method [8]. (d) The recolored image using Huang’s method [4]. (e) The recolored image using Khurge’s method [9]. (f)–(j) The corresponding images perceived by protanopia. (Color figure online)

Full size image

By contrast, our proposed method only recolors unrecognizable colors sequentially and take the differences between one unrecognizable color and other colors with higher confidence as the stopping iteration benchmark, which can make sure the differences after recoloring is no lower than before. As shown in Figs. 3(g), 4(g) and 5(g), only unrecognizable colors have been recolored, and the contrast between different colors is higher for human with protanopia than other methods.

Fig. 6.
figure 6

(a) The original Ishihara test image. (b) The recolored image using our proposed method. (c) The recolored image using Doliotis’s method [8]. (d) The recolored image using Huang’s method [4]. (e) The recolored image using Khurge’s method [9]. (f)–(j) The corresponding images perceived by protanopia.

Full size image

Because the Ishihara test images are used in medical to diagnose whether a person is CVD, we also conduct a series of experiments using some Ishihara test images and the experimental results are shown in Figs. 6 and 7. Similar to the natural image, our proposed method obtains the highest contrast between different colors for human with CVD than other methods.

Fig. 7.
figure 7

(a) The original Ishihara test image. (b) The recolored image using our proposed method. (c) The recolored image using Doliotis’s method [8]. (d) The recolored image using Huang’s method [4]. (e) The recolored image using Khurge’s method [9]. (f)–(j) The corresponding images perceived by protanopia.

Full size image

3.2 Objective Quality Comparison

We now objectively assess the quality of the recolored images for human with CVD using \(E_{contrast}\) which represents the average contrast between arbitrary pixel pairs. A higher score indicates better performance. The contrast calculation for one image is defined in Eq. (6).

$$\begin{aligned} E_{contrast}=\sum _{i}\sum _{j=i+1}D_{ij}/N \end{aligned}$$
(6)

where \(D_{ij}\) is the calculated difference between i-th and j-th pixels using (2) and N is the total number of the pixel pairs in the recolored image.

The \(E_{contrast}\) of the various recolored images perceived by CVD are listed in Table 1. As it is shows, our proposed method achieves highest contrast than other algorithms. Compared with original image, Doliotis [8], Huang [4] and Khurge [9], our proposed algorithm increases the average \(E_{contrast}\) by 6.56%, 21.80%, 9.4% and 20.49%, respectively.

Table 1. The \(E_{contrast}\) obtained by different algorithms
Full size table

3.3 Execution Speed Comparison

In this subsection, we compare the execution time of various algorithms. As shown in Table 2, Huang [4] requires more time to search optimal recolored colors for CVD than Doliotis [8], Khurge [9] and our proposed rule-based method. In addition, our proposed method further improves the execution time compared with Doliotis [8] because we judge and recolor unrecognizable colors sequentially instead of simultaneously. The execution time of our proposed method is approximately 5.4 times faster than Doliotis [8]. Khurge [9] recolors all pixels in the image by one multiplication operation, and our proposed method only recolors unrecognizable key colors. Therefore, if there exist a little unrecognizable colors in an image, our proposed method can execute faster than Khurge. For example, there exist only one unrecognizable color in the Fig. 4, so the execution time of our proposed method is less than Khurge.

Table 2. The execution speed of various algorithms (s)
Full size table

4 Conclusion

In this paper, an efficient recoloring method is proposed for color vision deficiency. To better adapt to the recoloring task, we improve the rule-based recoloring methods from three aspects. First, key color confidence is introduced to judge and recolor unrecognizable colors sequentially, which can speed up the execution time greatly. Second, differences between one color and other colors with higher confidence are taken as the unrecognizable benchmark for this color, which cannot involve any pre-defined parameters. Third, these differences are also used in iterative recoloring to make sure the contrast between different colors after recoloring is no lower than before. Experimental results show that our proposed method can recolor unrecognizable colors for CVD effectively.