Skip to main content
SpringerLink
Log in

Convolutional Neural Networks for Geo-Localisation with a Single Aerial Image

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Nowadays, Unmanned Aerial Vehicles (UAVs) navigating outdoors rely heavily on GPS for localisation and autonomous flight or applications for aerial photography re-cording with a GPS coordinate. However, GPS may fail or become unreliable, thus compromising the flight mission. Motivated by this scenario, in this work, we present a study on the use of popular Convolutional Neural Networks (CNN) to address the problem of geo-localisation from a single aerial image. We compare CNN-based architectures from the state-of-the-art, and introduce a compact architecture to speed up the inference process without affecting the estimation error. For our experiments, aerial images were recorded with a monocular camera onboard a UAV, flying outdoors with a height between 20 to 25 metres. On average, our compact network achieves a minimum estimation error of 2.8 metres and a maximum of 6.1 metres, which is comparable to the performance of other networks in the state-of-the-art. However, our network achieves on average an operation frequency of 103 fps versus 69 fps achieved by the fastest network in the comparison analysis. These results are promising since such speed would enable fast geo-localisation with cameras capturing images at those frame rates, which are useful for obtaining neater images than with conventional cameras working at 30 fps.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Meinen, B.U., Robinson, D.T.: Streambank topography: an accuracy assessment of uav-based and traditional 3d reconstructions. Int. J. Remote Sens. 41(1), 1–18 (2020)

    Article  Google Scholar 

  2. Coelho Eugenio, F., Badin, T.L., Fernandes, P., Mallmann, C.L., Schons, C., Schuh, M.S., Soares Pereira, R., Fantinel, R.A., Pereira da Silva, S.D.: Remotely piloted aircraft systems (rpas) and machine learning: a review in the context of forest science. Int. J. Remote Sens. 42(21), 8238–8266 (2021)

    Article  Google Scholar 

  3. Yang, N., Yang, S., Cui, W., Zhang, Z., Zhang, J., Chen, J., Ma, Y., Lao, C., Song, Z., Chen, Y.: Effect of spring irrigation on soil salinity monitoring with uav-borne multispectral sensor. Int J Remote Sens (2021)

  4. Ham, Y., Han, K.K., Lin, J.J., Golparvar-Fard, M.: Visual monitoring of civil infrastructure systems via camera-equipped unmanned aerial vehicles (uavs): a review of related works. Vis. Eng. 4(1), 1–8 (2016)

    Article  Google Scholar 

  5. Tripolitsiotis, A., Prokas, N., Kyritsis, S., Dollas, A., Papaefstathiou, I., Partsinevelos, P.: Dronesourcing: a modular, expandable multi-sensor uav platform for combined, real-time environmental monitoring. Int. J. Remote Sens. 38(8–10), 2757–2770 (2017)

    Article  Google Scholar 

  6. Cantieri, A., Ferraz, M., Szekir, G., Antônio Teixeira, M., Lima, J., Schneider Oliveira, A., Aurélio Wehrmeister, M.: Cooperative uav-ugv autonomous power pylon inspection: an investigation of cooperative outdoor vehicle positioning architecture. Sensors 20(21), 6384 (2020)

    Article  Google Scholar 

  7. Parlange, R., Martinez-Carranza, J.: Leveraging single-shot detection and random sample consensus for wind turbine blade inspection. Intelligent Service Robotics pp. 1–18 (2021)

  8. Bodó, Z., Lantos, B.: State estimation for uavs using sensor fusion. In: 2017 IEEE 15th International Symposium on Intelligent Systems and Informatics (SISY), pp. 000111–000116 (2017). https://doi.org/10.1109/SISY.2017.8080535

  9. Davison, A.J., Reid, I.D., Molton, N.D., Stasse, O.: Monoslam: Real-time single camera slam. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 1052–1067 (2007)

    Article  Google Scholar 

  10. Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: Orb-slam: a versatile and accurate monocular slam system. IEEE Trans. Robot. 31(5), 1147–1163 (2015)

    Article  Google Scholar 

  11. Mur-Artal, R., Tardós, J.D.: Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Trans. Robot. 33(5), 1255–1262 (2017)

    Article  Google Scholar 

  12. Martinez-Carranza, J., Bostock, R., Willcox, S., Cowling, I., Mayol-Cuevas, W.: Indoor mav auto-retrieval using fast 6d relocalisation. Adv. Robot. 30(2), 119–130 (2016)

    Article  Google Scholar 

  13. Kendall, A., Grimes, M., Cipolla, R.: Posenet: A convolutional network for real-time 6-dof camera relocalization. In: Proceedings of the IEEE international conference on computer vision, pp. 2938–2946 (2015)

  14. Xiaogang, R., Wenjing, Y., Jing, H., Peiyuan, G., Wei, G.: Monocular depth estimation based on deep learning:a survey. In: 2020 Chinese Automation Congress (CAC), pp. 2436–2440 (2020). https://doi.org/10.1109/CAC51589.2020.9327548

  15. Pellegrin, L., Martinez-Carranza, J.: Towards depth estimation in a single aerial image. Int. J. Remote Sens. 41(5), 1970–1985 (2020)

    Article  Google Scholar 

  16. Lopez-Campos, R., Martinez-Carranza, J.: Espada: Extended synthetic and photogrammetric aerial-image dataset. IEEE Robot. Autom. Lett. 6(4), 7981–7988 (2021)

    Article  Google Scholar 

  17. Osuna-Coutiño, J.J., Martinez-Carranza, J.: Structure extraction in urbanized aerial images from a single view using a cnn-based approach. Int. J. Remote Sens. 41(21), 8256–8280 (2020)

    Article  Google Scholar 

  18. Cabrera-Ponce, A.A., Martinez-Carranza, J.: Aerial geo-localisation for mavs using posenet. In: 2019 Workshop on Research, Education and Development of Unmanned Aerial Systems (RED UAS), pp. 192–198. IEEE (2019)

  19. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  20. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  21. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778 (2016)

  22. Arreola, L., de Oca, A.M., Flores, A., Sanchez, J., Flores, G.: Improvement in the uav position estimation with low-cost gps, ins and vision-based system: Application to a quadrotor uav. In: 2018 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 1248–1254. IEEE (2018)

  23. Garcia-Huerta, R.A., Villalon-Turrubiates, I.E., GonzcHez-Jíménez, L.E., Allende-Alba, G.: Accuracy estimation of a low-cost gps receiver using landmarks on aerial images. In: IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, pp. 9244–9247. IEEE (2019)

  24. Zhao, C., Fan, B., Hu, J., Pan, Q., Xu, Z.: Homography-based camera pose estimation with known gravity direction for uav navigation. Sci. China Inf. Sci. 64(1), 1–13 (2021)

    Article  Google Scholar 

  25. Conte, G., Doherty, P.: An integrated uav navigation system based on aerial image matching. In: 2008 IEEE Aerospace Conference, pp. 1–10. IEEE (2008)

  26. Helgesen, H.H., Leira, F.S., Bryne, T.H., Albrektsen, S.M., Johansen, T.A.: Real-time georeferencing of thermal images using small fixed-wing uavs in maritime environments. ISPRS J. Photogramm. Remote Sens. 154, 84–97 (2019)

    Article  Google Scholar 

  27. Ding, L., Zhou, J., Meng, L., Long, Z.: A practical cross-view image matching method between uav and satellite for uav-based geo-localization. Remote Sens. 13(1), 47 (2021)

    Article  Google Scholar 

  28. Zamir, A.R., Shah, M.: Image geo-localization based on multiple nearest neighbour feature matching using generalized graphs. IEEE Trans. Pattern Anal. Mach. Intell. 36(8), 1546–1558 (2014)

    Article  Google Scholar 

  29. Costea, D., Leordeanu, M.: Aerial image geolocalization from recognition and matching of roads and intersections. arXiv preprint arXiv:1605.08323 (2016)

  30. Chathuranga, T.S., Munasinghe, R.: Aerial image matching based relative localization of a uav in urban environments. In: 2019 Moratuwa Engineering Research Conference (MERCon), pp. 633–637. IEEE (2019)

  31. Chebrolu, N., Lottes, P., Läbe, T., Stachniss, C.: Robot localization based on aerial images for precision agriculture tasks in crop fields. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 1787–1793. IEEE (2019)

  32. Le, T., Gjevestad, J.G.O., From, P.J.: Online 3d mapping and localization system for agricultural robots. IFAC-PapersOnLine 52(30), 167–172 (2019)

    Article  Google Scholar 

  33. Milford, M.J., Wyeth, G.F.: Seqslam: Visual route-based navigation for sunny summer days and stormy winter nights. In: 2012 IEEE International Conference on Robotics and Automation, pp. 1643–1649. IEEE (2012)

  34. Jin, R., Jiang, J., Qi, Y., Lin, D., Song, T.: Drone detection and pose estimation using relational graph networks. Sensors 19(6), 1479 (2019)

    Article  Google Scholar 

  35. de Lima, R., Cabrera-Ponce, A.A., Martinez-Carranza, J.: Parallel hashing-based matching for real-time aerial image mosaicing. J. Real-Time Image Proc. 18(1), 143–156 (2021)

    Article  Google Scholar 

  36. Trigkakis, D., Petrakis, G., Tripolitsiotis, A., Partsinevelos, P.: Automated geolocation in urban environments using a simple camera-equipped unmanned aerial vehicle: A rapid mapping surveying alternative? ISPRS Int. J. Geo-Inf. 9(7), 425 (2020)

    Article  Google Scholar 

  37. Shetty, A., Gao, G.X.: Uav pose estimation using cross-view geolocalization with satellite imagery. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 1827–1833. IEEE (2019)

  38. von Stumberg, L., Wenzel, P., Yang, N., Cremers, D.: Lm-reloc: Levenberg-marquardt based direct visual relocalization. arXiv preprint arXiv:2010.06323 (2020)

  39. Winkelbauer, D., Denninger, M., Triebel, R.: Learning to localize in new environments from synthetic training data. arXiv preprint arXiv:2011.04539 (2020)

  40. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9 (2015)

  41. Kendall, A., Cipolla, R.: Modelling uncertainty in deep learning for camera relocalization. In: 2016 IEEE international conference on Robotics and Automation (ICRA), pp. 4762–4769. IEEE (2016)

  42. Kendall, A., Cipolla, R.: Geometric loss functions for camera pose regression with deep learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5974–5983 (2017)

  43. Seifi, S., Tuytelaars, T.: How to improve cnn-based 6-dof camera pose estimation. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0 (2019)

  44. Walch, F., Hazirbas, C., Leal-Taixe, L., Sattler, T., Hilsenbeck, S., Cremers, D.: Image-based localization using lstms for structured feature correlation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 627–637 (2017)

  45. Valada, A., Radwan, N., Burgard, W.: Deep auxiliary learning for visual localization and odometry. In: 2018 IEEE international conference on robotics and automation (ICRA), pp. 6939–6946. IEEE (2018)

  46. Blanton, H., Greenwell, C., Workman, S., Jacobs, N.: Extending absolute pose regression to multiple scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 38–39 (2020)

  47. Zhang, R., Luo, Z., Dhanjal, S., Schmotzer, C., Hasija, S.: Posenet++: A cnn framework for online pose regression and robot re-localization

  48. Bresson, G., Li, Y., Joly, C., Moutarde, F.: Urban localization with street views using a convolutional neural network for end-to-end camera pose regression (2019)

  49. Kadosh, M., Moses, Y., Shamir, A.: On the role of geometry in geo-localization. arXiv preprint arXiv:1906.10855 (2019)

  50. Kim, H.J., Dunn, E., Frahm, J.M.: Learned contextual feature reweighting for image geo-localization. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3251–3260. IEEE (2017)

  51. Workman, S., Souvenir, R., Jacobs, N.: Wide-area image geolocalization with aerial reference imagery. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3961–3969 (2015)

  52. Sun, B., Chen, C., Zhu, Y., Jiang, J.: Geocapsnet: Aerial to ground view image geo-localization using capsule network. arXiv preprint arXiv:1904.06281 (2019)

  53. Altwaijry, H., Veit, A., Belongie, S.J., Tech, C.: Learning to detect and match keypoints with deep architectures. In: BMVC (2016)

  54. Altwaijry, H., Trulls, E., Hays, J., Fua, P., Belongie, S.: Learning to match aerial images with deep attentive architectures. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3539–3547 (2016)

  55. Lin, T.Y., Cui, Y., Belongie, S., Hays, J.: Learning deep representations for ground-to-aerial geolocalization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5007–5015 (2015)

  56. Müller, M., Urban, S., Jutzi, B.: Squeezeposenet: Image based pose regression with small convolutional neural networks for real time uas navigation. ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. 4, 49 (2017)

    Article  Google Scholar 

  57. Amer, K., Samy, M., Shaker, M., ElHelw, M.: Deep convolutional neural network-based autonomous drone navigation. arXiv preprint arXiv:1905.01657 (2019)

  58. Hu, S., Chang, X.: Multi-view drone-based geo-localization via style and spatial alignment. arXiv preprint arXiv:2006.13681 (2020)

  59. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818–2826 (2016)

  60. Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al.: Recent advances in convolutional neural networks. Pattern Recogn. 77, 354–377 (2018)

    Article  Google Scholar 

  61. Ahmadi, M., Vakili, S., Langlois, J.P., Gross, W.: Power reduction in cnn pooling layers with a preliminary partial computation strategy. In: 2018 16th IEEE International New Circuits and Systems Conference (NEWCAS), pp. 125–129. IEEE (2018)

  62. Cabrera-Ponce, A.A., Martin-Ortiz, M., Martinez-Carranza, J.: Continual learning for multi-camera relocalisation. In: Mexican International Conference on Artificial Intelligence, pp. 289–302. Springer (2021)

Download references

Acknowledgements

The first author is thankful for his scholarship funded by Consejo Nacional de Ciencia y Tecnología (CONACYT) under the grant 727018.

Author information

Authors and Affiliations

  1. INAOE, San Andrés Cholula, Mexico

    Aldrich A. Cabrera-Ponce & Jose Martinez-Carranza

Authors
  1. Aldrich A. Cabrera-Ponce
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jose Martinez-Carranza
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jose Martinez-Carranza.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cabrera-Ponce, A.A., Martinez-Carranza, J. Convolutional Neural Networks for Geo-Localisation with a Single Aerial Image. J Real-Time Image Proc 19, 565–575 (2022). https://doi.org/10.1007/s11554-022-01207-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-022-01207-1

Keywords

Search

Navigation