Siamese trackers have received a lot of attentions due to their promising performance and real-time high speed. However, the robustness is generally limited especially in challenging conditions, such as occlusion. Motivated by that tracking failure does not always occur and the target movement tends to follow certain patterns, in the paper, we propose a generic, fast and flexible approach to improve the robustness of Siamese trackers with two light-load novel modules: Trajectory Guidance Module (TGM) and Selective Refinement Module (SRM). Specifically, TGM encourages to pay a soft attention on possible target location based on short-term historical trajectory. SRM selectively remedies the tracking results at the risk of failure with little impact on the speed. The proposed algorithm can be easily establish upon state-of-the-art Siamese trackers and obtains better performance on seven benchmarks with high real-time tracking speed. The code is available at https://github.com/TJUMMG/TGSR.