Color is important for humans to receive visual information such as from traffic lights and maps. However sometimes important color-based information cannot be correctly received by some visually impaired people such as protanopes and deuteranopes. It is necessary to understand their difficulties in perceiving color and provide solutions to improve their quality of life. In our paper, a lightness modification method that can convey the color-based information to protanopes and deuteranopes is proposed. By considering color differences in an input image, the proposed method modifies the output lightness of an image for protanopes and deuteranopes to preserve its visual detail. The proposed method only changes the lightness without changing the hue, resulting in the output image having natural colors. In experiments, we use six color blindness test images with different color distributions and compare the proposed method with three existing methods. The experiment results and their quantitative evaluation show that our proposed method is reliable and effective.