Visual trackers based on the framework of kernelized correlation filter (KCF) need to learn information on the object from each frame, thus the state change of the object affects the tracking performances. In this paper, we propose a novel KCF tracker using luminance histogram and adaptive model, in order to deal with the change of the object’s state. This method firstly takes skipped scale pool method which utilizes variable window size at every two frames. Secondly, the location of the object is estimated using the combination of the filter response and the similarity of the luminance histogram at multiple points in the filter response map. Thirdly, the learning rate to obtain the tracking model is adjusted, using the filter response and the similarity of the luminance histogram, considering the state of the object. Experimentally, the proposed tracker (CFHA) achieves outstanding performance for the challenging benchmark sequence (OTB100).