Traffic light detection and recognition play a more important role in Advanced Driver Assistance Systems and driverless cars. This paper presents a method of integrating Visual Selective Attention (VSA) model with HOG features to solve the problem of detecting and recognizing traffic lights in complex urban environment. First of all, the VSA model is used to get candidate regions of the traffic lights. Then, the HOG features of the traffic lights and SVM classifier are used in these candidate regions to get precise regions of traffic lights. Within these regions, the color of traffic light is recognized according to the information in the gray-scale image of channel A. Experimental results show that the proposed method has strong robustness and high accuracy.