Template tracking is a typical paradigm to adaptively locate arbitrary objects in the tracking literature. Although existing works present diverse template updating approaches, one of the essential problems of template updating has not been solved effectively, i.e., when and how to update a template. In this work, we treat the updating time as an abnormal moment that indicates the previous template cannot depict the target accurately any more. Thus, we introduce an effective State-Edge Awareness (SEA) module that detect such abnormal moments via unsupervised anomaly detection. To be specific, by retaining multi search frames of a video, SEA firstly analysis the correlation features that generated by the template and search images. Then, it estimates the measurement for abnormal degree that is regarded as the sign for template updating. As a result, our method can not only capture the updating time automatically, but also update the templates effectively. Furthermore, the effectiveness of the proposed method has been verified on a representative CNN-based and Transformer-based tracker, respectively. The experimental results on five popular benchmarks show that our tracker can achieve the state-of-the-art performance.