Unsharp masking is a popular image enhancement technique used to sharpen the image appearance in gray images. When applied to the color image, unsharp masking tends to introduce color distortion. In this paper, we propose a novel unsharp masking technique which can sharpen the image appearance without distorting the color and amplifying the noise. The proposed algorithm exploits the characteristics of the human visual system to control the level of sharpening and uses the energy dispersion filter to remove the color distortion. Experimental results show that our proposed algorithm successfully produces sharpened images that are free of noise and color distortion commonly found in conventional unsharp masking algorithm.