The autonomous flight of unmanned aerial vehicles (UAVs) relies on a precise and robust geo-localization system. A visual geo-localization system registers the aerial captured image to a geo-referenced satellite map, which enables UAVs to determine their global position without a global navigation satellite system (GNSS). However, it is challenging to achieve precise and robust geo-localization in a large-scale environment due to the notable texture difference between the satellite map and the UAV-captured image. In this article, we design a robust visual geo-localization pipeline that integrates a proposed deep learning-based imagery feature. This pipeline starts with the image retrieval based on the deep feature encoding, to initialize the localization process over large-scale maps without any location prior. With the reuse of the same deep imagery feature, an image registration process enables real-time sequential localization. Besides, the proposed system has the re-localization ability to eliminate the localization drift caused by possible registration failure, especially during long-time flights. Evaluations of datasets and real-world experiments demonstrate that the proposed system is more robust and accurate than other state-of-the-art methods. To encourage further progress on the visual geo-localization problem, our code and materials are publicly available at https://github.com/hmf21/UAVLocalization.