Many tracking problems can be efficiently solved by the Altering technique. Linear filter methods (e.g. Kaiman Filter) have shown their success and optimally in many linear settings with Gaussian noises. However, they expose inefficiency and weakness in the general nonlinear and high dimensional setting (e.g. human tracking). While, the advancement of Sequential Importance Re-sampling with Simulated Annealing has shown it is capable of handling nonlinearity and high dimensionality of human tracking. However, its performance is often affected by lighting variations and noises from silhouette segmentation. The proposed approach incorporates a textured human body template to annealed sequential filtering, and uses the illumination invariant CIELab formula to evaluate the observation likelihood so that influences of lighting changes and noises are minimised. Experiments with the benchmark HumanEval dataset demonstrate encouraging improvements over traditional Sequential Importance Re-sampling and the silhouette based method.