Tracking objects or humans using background subtraction is commonly used both in indoor and outdoor environment. But this method may not produce quality results due to illumination change, shadows, and occlusion occurred in indoor or outdoor environments. Song and Nevatia suggested a simultaneous tracking of face and body for human tracking in an indoor environment using Adaboost face detection and Camshift color tracking methods (X. Song and R. Nevatia, 2004). Changes in illumination are not considered in their work. In this research, an intelligent method for color tracking scheme using skin reflectance values is proposed to track the human in real-time video for varying illuminations both in indoor and outdoor environments. This research finds application in the live transmission of an indoor video conference or a live news cast from an outdoor as well as in the places where the illumination change occurs frequently.