Correlation filter (CF)-based approaches have gained widespread attention in the field of unmanned aerial vehicle (UAV) visual tracking due to their light-weight characteristics. However, CFs are prone to generating low-quality response in challenging UAV scenarios, e.g., fast motion and background clutter. In this paper, in order to model the tracker more robustly, we first conduct an effective regularization analysis from the perspectives of response- and background-learning. Specifically, to address response degradation, we propose a module for learning temporal consistency and reversibility of response, supplemented by a novel background-aware module to enhance the ability to learn from negative samples. In addition, we propose a fast coarse-to-fine scale search strategy, which alleviates the challenges in estimating bounding boxes under non-uniform aspect ratios. We have developed two tracker versions, namely RBLT and DeepRBLT, based on the depth of the features. Comprehensive experiments on four UAV benchmarks and one generic benchmark have indicated the superiority of our trackers compared to other state-of-the-art trackers, with enough speed for real-time applications.