In this paper, we have developed a Gaussian Mixture Model (GMM) based algorithm with dynamic patch estimation for real-time detection and tracking of a known object. This research work detects the object of interest, estimates its 3-D position using Extended Kalman Filter (EKF) and generates the control output to the quad-rotor to track the target. The proposed algorithm is capable of tracking the object with a high Frame Per Second (FPS). Rigorous experiments are carried out to demonstrate the efficacy of the proposed approach in outdoor environment.