Correlation filter (CF) has received increasing attention in online visual object tracking. By accelerating the correlation in the frequency domain, CF trackers have achieved superior performance. However, existing CF trackers have a common shortcoming in that tracking models are prone to drift due to error accumulation. To address this problem, we propose a two-stage cascaded framework for accurate object modeling. In the first stage, we propose a passive–aggressive correlation filter (PACF) tracker to reduce error accumulation. In the subsequent stage, an online refinement algorithm is used to calibrate the tracking model by exploiting both long-term and short-term cues. In order to achieve high efficiency, our scheme reuses the PACF tracking response in the following stage. Extensive experiments were conducted on both long-term and short-term visual tracking benchmarks. The experimental results demonstrate that our tracker outperforms the state-of-the-art online tracking schemes in both long-term and short-term settings. Finally, we present a comprehensive analysis to validate the efficacy of our proposed method.