In this paper, a vision tracking system via color detection is presented. In this system, a monocular camera is used as a sensor to track and measure the relative pose of the target based on specified color detection with the help of four LED markers. Kalman filter is used in this system to predict the position of LED markers in the next frame of the camera image so as to decrease the area in which to search for the markers, it decreases the possibility of wrong detection. An (unmanned aerial vehicle) UAV is used as a platform to test the effectiveness of this system. Experimental results show that the proposed system can track the target (UAV) in a constant speed very well.