Self-localization is of paramount importance for autonomous vehicles, since the system interprets traffic scene context with a combination of high definition map and a precise ego-pose. Therefore, alerting the driver of a potential failure ahead of the actual localization failure, is an essential function for any autonomous driving system. This paper introduces a Time-to-Failure (TTF) concept in the localization domain. We propose a TTF predictor with a ResNet34-based feature extractor followed by a LSTM-based regressor. In order to train the predictor, an efficient training data generation scheme using simulation with intentional noises, is also shown. Evaluation of the proposed method is done in the context of regression and classification. The preliminary experimental results show that the proposed method can predict localization failure by up to 10 seconds ahead of the actual event.