This paper presents a robust histogram based tracking algorithm that is capable of detecting specular highlights on objects or faces to be tracked. The materials with shiny and smooth surfaces such as car exterior, ceramics or glasses often exhibit specularities which are highly saturated regions in the image that are produced by mirrorlike reflections. Whenever confronted with such specular highlights on the target objects, the results of segmentation and tracking become inaccurate and unreliable. Speaking of real-time color object tracking, there are two major issues that are associated with such specular highlights. First issue is how to detect specular highlights suddenly appearing in the image sequence. Second one is how the detected specular highlights can be correspondingly considered to improve the tracking performance. In this paper, we describe our specularity detection method that can he applied to every pair of consecutive frames in the tracking sequence. Experimental results of two tracking systems: (1) with specularity detection and (2) without handling specularities are compared to show the improvement. This method has been successfully tested on multiple tracking tasks with monochromatic objects.