Cross view image-based geo-localization aims to estimate global position of an image by matching the query image with the images in a geo-referenced image database. Cross-view image-based geo-localization is a potential supplement of GPS, but it's hard to matching cross-view image pairs because of the tremendous appearance differences. However, most of deep learning approaches match cross-view image pairs directly and ignore the inner relation between them. In this paper, we introduce our novel hybrid perspective mapping method to align ground-level image to aerial image by considering projection relation between them. Unlike other learning based method which generate corresponding aerial image by traning, our approach is totally geometry based and can be plugged to other network conveniently. And we propose our network based on hybrid perspective mapping. our network shows higher retrieval accuracy and powerful generalization ability on several public dataset. In addition, we also conduct cross-view image matching experiments on our own dataset and analyse the influence of spatial resolution and seasonal variation of aerial image on image matching.