Color-to-gray image conversion is commonly used in applications such as printing, e-ink display, image stylization. Effective decolorization methods aim to maintain the naturalness and the image contrast after the conversion. We propose a method for this conversion problem using the image’s salient colors and radial basis functions. There are three steps in the proposed method. First, we quantize the image colors to a small number of salient representative colors, which essentially provides an adaptive mechanism for choosing the dominant image color regime that can be mapped to the grayscale with the maximum possible contrast structure. A carefully designed criterion is developed for choosing the appropriate number of quantization points in the k-means algorithm based on the elbow method. Second, two methods are proposed for ordering the quantized colors on the grayscale intensities, using the guideline that the ordering should agree with the human perceptual comfort when looking at common grayscale images. Finally, radial basis functions are used to extend the mapping from the quantized colors to the grayscale intensities to all colors of the original image’s pixels in a continuous fashion. Experimental results on the three benchmark datasets of color-to-gray conversion show significant performance improvement over state-of-the-art methods both qualitatively and quantitatively.