Recently, visual trackers based on the framework of kernelized correlation filter (KCF) achieve the robustness and accuracy results. These trackers need to learn information on the object from each frame, thus the state change of the object affects the tracking performances. In order to deal with the state change, we propose a novel KCF tracker using the filter response map, namely a confidence map, and adaptive model. This method firstly takes a skipped scale pool method which utilizes variable window size at every two frames. Secondly, the location of the object is estimated using the combination of the filter response and the similarity of the luminance histogram at multiple points in the confidence map. Moreover, we use the re-detection of the multiple peaks of the confidence map to prevent the target drift and reduce the influence of illumination. Thirdly, the learning rate to obtain the model of the object is adjusted, using the filter response and the similarity of the luminance histogram, considering the state of the object. Experimentally, the proposed tracker (CFCA) achieves outstanding performance for the challenging benchmark sequence (OTB2013 and OTB2015).