Color is important for the visual information. However, sometimes the information of color cannot be correctly received by some humans. Those visually impaired humans (protanopes, deuteranopes, and so on) cannot recognize some color information. In our paper, a lightness modification method which can convey the information of color to protanopes and deuteranopes is proposed. Considering differences of colors in an input image, the proposed method adjusts the output lightness for protanopes and deuteranopes. Experimental results show effectiveness of our proposed method.