Skip to main content
SpringerLink
Log in

Urban localization based on aerial imagery by correcting projection distortion

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

This study proposes a vehicle localization method that fuses aerial maps and LiDAR measurements in urban canyon environments. The building outlines from an aerial image can be used as appropriate features for matching with the LiDAR data for localization. However, distortions caused by scaled orthographic projection of aerial maps are commonly observed in the images of metropolitan areas, which may significantly degrade the matching and resulting localization performance. In this study, a novel method for correcting such distortions is proposed and used for the vehicle localization by matching the corrected map and LiDAR measurements. Instance and semantic segmentation algorithms were used to distinguish individual buildings and generate corrected outlines of the buildings. A particle filter is applied to determine the pose of the vehicle based on the mutual information between the map and LiDAR measurements. The performance of the proposed algorithm was verified using a dataset obtained in urban areas.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Alharake, A., Bresson, G., Merriaux, P., Vauchey, V., & Savatier, X. (2019). Urban localization inside cadastral maps using a likelihood field representation. In 2019 IEEE intelligent transportation systems conference (ITSC) (pp. 1329–1335). https://doi.org/10.1109/itsc.2019.8917303

  • Audebert, N., Le Saux, B., & Lefèvre, S. (2018). Beyond RGB: Very high resolution urban remote sensing with multimodal deep networks. ISPRS Journal of Photogrammetry and Remote Sensing, 140, 20–32. https://doi.org/10.1016/j.isprsjprs.2017.11.011

    Article  Google Scholar 

  • Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481–2495.

    Article  Google Scholar 

  • Cai, S., Guo, Y., Khan, S., Hu, J., & Wen, G. (2019). Ground-to-aerial image geolocalization with a hard exemplar reweighting triplet loss. In Proceedings of the IEEE international conference on computer vision (pp. 8391–8400). https://doi.org/10.1109/iccv.2019.00848

  • Chen, H., Zhang, K., Xiao, W., Sheng, Y., Cheng, L., Zhou, W., & Zhang, S. (2021). Building change detection in very high-resolution remote sensing image based on pseudo-orthorectification. International Journal of Remote Sensing, 42(7), 2686–2705. https://doi.org/10.1080/01431161.2020.1862437

    Article  Google Scholar 

  • Chu, H., Mei, H., Bansal, M., & Walter, M. R. (2015). Accurate vision-based vehicle localization using satellite imagery. arXiv preprint arXiv:1510.09171

  • Cover, T. M., & Thomas, J. A. (2006). Elements of information theory (2nd ed., p. 13). John Wiley & Sons. https://doi.org/10.1002/047174882X

    Book  MATH  Google Scholar 

  • de Paula Veronese, L., de Aguiar, E., Nascimento, R. C., Guivant, J., Cheein, F. A. A., De Souza, A. F., & Oliveira-Santos, T. (2015). Reemission and satellite aerial maps applied to vehicle localization on urban environments. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 4285–4290). https://doi.org/10.1109/iros.2015.7353984

  • D’Errico, J. (2019). A suite of minimal bounding objects. https://kr.mathworks.com/ matlabcentral/fileexchange/34767.

  • de Veronese, L. P., Auat-Cheein, F., Mutz, F., Oliveira-Santos, T., Guivant, J. E., de Aguiar, E., & De Souza, A. F. (2020). Evaluating the limits of a LiDAR for an autonomous driving localization. IEEE Transactions on Intelligent Transportation Systems. https://doi.org/10.1109/tits.2020.2971054

    Article  Google Scholar 

  • Fang, Y., Wang, C., Yao, W., Zhao, X., Zhao, H., & Zha, H. (2019). On-road vehicle tracking using part-based particle filter. IEEE Transactions on Intelligent Transportation Systems, 20(12), 4538–4552.

    Article  Google Scholar 

  • Floros, G., Van Der Zander, B., & Leibe, B. (2013). Openstreetslam: Global vehicle localization using openstreetmaps. In 2013 IEEE International conference on robotics and automation (pp. 1054–1059). https://doi.org/10.1109/icra.2013.6630703

  • Früh, C., & Zakhor, A. (2004). An automated method for large-scale, ground-based city model acquisition. International Journal of Computer Vision, 60(1), 5–24. https://doi.org/10.1023/b:visi.0000027787.82851.b6

    Article  Google Scholar 

  • Fu, M., Zhu, M., Yang, Y., Song, W., & Wang, M. (2020). LiDAR-based vehicle localization on the satellite image via a neural network. Robotics and Autonomous Systems. https://doi.org/10.1016/j.robot.2020.103519

    Article  Google Scholar 

  • Guiasu, S. (1977). Information theory with applications (Vol. 202). McGraw-Hill.

    MATH  Google Scholar 

  • Han, J., Cho, Y., Kim, J., Kim, J., Son, N.-S., & Kim, S. Y. (2020). Autonomous collision detection and avoidance for ARAGON USV: Development and field tests. Journal of Field Robotics, 37(6), 987–1002. https://doi.org/10.1002/rob.21935

    Article  Google Scholar 

  • Han, J., Kim, J., & Shim, D. H. (2018). Precise localization and mapping in indoor parking structures via parameterized SLAM. IEEE Transactions on Intelligent Transportation Systems, 20(12), 4415–4426. https://doi.org/10.1109/tits.2018.2885341

    Article  Google Scholar 

  • Han, J., Park, J., Kim, T., & Kim, J. (2015). Precision navigation and mapping under bridges with an unmanned surface vehicle. Autonomous Robots, 38(4), 349–362. https://doi.org/10.1007/s10514-015-9419-2

    Article  Google Scholar 

  • He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask R-CNN. In Proceedings of the IEEE international conference on computer vision (pp. 2961–2969). https://doi.org/10.1109/iccv.2017.322

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778). https://doi.org/10.1109/cvpr.2016.90

  • Hu, H., Zhu, J., Wirges, S., & Lauer, M. (2019). Localization in aerial imagery with grid maps using LocGAN. In 2019 IEEE intelligent transportation systems conference (ITSC) (pp. 2860–2865). https://doi.org/10.1109/itsc.2019.8917236

  • Hu, S., & Lee, G. H. (2020). Image-based geolocalization using satellite imagery. International Journal of Computer Vision, 128(5), 1205–1219. https://doi.org/10.1007/s11263-019-01186-0

    Article  Google Scholar 

  • Javanmardi, M., Javanmardi, E., Gu, Y., & Kamijo, S. (2017). Towards high-definition 3D urban mapping: Road feature-based registration of mobile mapping systems and aerial imagery. Remote Sensing, 9(10), 975. https://doi.org/10.3390/rs9100975

    Article  Google Scholar 

  • Jende, P., Nex, F., Gerke, M., & Vosselman, G. (2018). A fully automatic approach to register mobile mapping and airborne imagery to support the correction of platform trajectories in GNSS-denied urban areas. ISPRS Journal of Photogrammetry and Remote Sensing, 141, 86–99. https://doi.org/10.1016/j.isprsjprs.2018.04.017

    Article  Google Scholar 

  • Jeong, J., Cho, Y., Shin, Y.-S., Roh, H., & Kim, A. (2019). Complex urban dataset with multi-level sensors from highly diverse urban environments. The International Journal of Robotics Research. https://doi.org/10.1177/0278364919843996

    Article  Google Scholar 

  • Kampffmeyer, M., Salberg, A.-B., & Jenssen, R. (2016). Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks. In IEEE conference on computer vision and pattern recognition workshops (pp. 1–9). https://doi.org/10.1109/cvprw.2016.90

  • Käslin, R., Fankhauser, P., Stumm, E., Taylor, Z., Mueggler, E., Delmerico, J., & Hutter, M. (2016). Collaborative localization of aerial and ground robots through elevation maps. In IEEE international symposium on safety, security, and rescue robotics (pp. 284–290). https://doi.org/10.1109/ssrr.2016.7784317

  • Kim, D.-K., & Walter, M. R. (2017). Satellite image-based localization via learned embeddings. In IEEE international conference on robotics and automation (pp. 2073–2080). https://doi.org/10.1109/icra.2017.7989239

  • Kim, J., Cho, Y., & Kim, J. (2018). Vehicle localization in urban environment using a 2d online map with building outlines. In 2018 15th International conference on ubiquitous robots (UR) (pp. 586–590). https://doi.org/10.1109/urai.2018.8441821

  • Kim, J., & Kim, J. (2019). Fusing lidar data and aerial imagery with perspective correction for precise localization in urban canyons. In 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 5298–5303). https://doi.org/10.1109/iros40897.2019.8967711

  • Kümmerle, R., Steder, B., Dornhege, C., Kleiner, A., Grisetti, G., & Burgard, W. (2011). Large scale graph-based SLAM using aerial images as prior information. Autonomous Robots, 30(1), 25–39. https://doi.org/10.1007/s10514-010-9204-1

    Article  Google Scholar 

  • Landsiedel, C., & Wollherr, D. (2017). Global localization of 3d point clouds in building outline maps of urban outdoor environments. International Journal of Intelligent Robotics and Applications, 1(4), 429–441. https://doi.org/10.1007/s41315-017-0038-2

    Article  Google Scholar 

  • Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2117–2125). https://doi.org/10.1109/cvpr.2017.106

  • Liu, L., & Li, H. (2019). Lending orientation to neural networks for cross-view geolocalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5624–5633). https://doi.org/10.1109/cvpr.2019.00577

  • NGII (2019). National geographic information institute. www.ngii.go.kr

  • Roh, H., Jeong, J., Cho, Y., & Kim, A. (2016). Accurate mobile urban mapping via digitalmap-based SLAM. Sensors, 16(8), 1315. https://doi.org/10.3390/s16081315

    Article  Google Scholar 

  • Roh, H., Jeong, J., & Kim, A. (2017). Aerial image based heading correction for large scale SLAM in an urban canyon. IEEE Robotics and Automation Letters, 2(4), 2232–2239. https://doi.org/10.1109/lra.2017.2725439

    Article  Google Scholar 

  • Ruchti, P., Steder, B., Ruhnke, M., & Burgard, W. (2015). Localization on openstreetmap data using a 3d laser scanner. In 2015 IEEE international conference on robotics and automation (ICRA) (pp. 5260–5265). https://doi.org/10.1109/icra.2015.7139932

  • Senlet, T., & Elgammal, A. (2011). A framework for global vehicle localization using stereo images and satellite and road maps. In IEEE international conference on computer vision workshops (pp. 2034–2041). https://doi.org/10.1109/iccvw.2011.6130498

  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556

  • Tang, T. Y., De Martini, D., Wu, S., & Newman, P. (2020). Self-supervised localisation between range sensors and overhead imagery. arXiv preprint arXiv:2006.02108

  • Tian, Y., Chen, C., & Shah, M. (2017). Cross-view image matching for geo-localization in urban environments. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3608–3616). https://doi.org/10.1109/cvpr.2017.216

  • Viswanathan, A., Pires, B. R., & Huber, D. (2016). Vision-based robot localization across seasons and in remote locations. In 2016 IEEE international conference on robotics and automation (ICRA) (pp. 4815–4821). https://doi.org/10.1109/icra.2016.7487685

  • Vora, A., Agarwal, S., Pandey, G., & McBride, J. (2020). Aerial imagery based LIDAR localization for autonomous vehicles. arXiv preprint arXiv:2003.11192

  • Vysotska, O., & Stachniss, C. (2016). Exploiting building information from publicly available maps in graph-based slam. In 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 4511–4516). https://doi.org/10.1109/iros.2016.7759664

  • Wang, X., Vozar, S., & Olson, E. (2017). Flag: Feature-based localization between air and ground. In 2017 IEEE international conference on robotics and automation (ICRA) (pp. 3178–3184). https://doi.org/10.1109/icra.2017.7989360

  • Yoo, T.-S., Kim, M.-H., Yoon, S.-I., & Kim, D.-J. (2019). Design of tightly coupled INS/DVL/RPM integrated navigation system. Journal of Ocean Engineering and Technology, 33(5), 470–478. https://doi.org/10.26748/ksoe.2019.085

    Article  Google Scholar 

  • Zhang, K., Sheng, Y., Wang, M., & Fu, S. (2018). An enhanced multi-view vertical line locus matching algorithm of object space ground primitives based on positioning consistency for aerial and space images. ISPRS Journal of Photogrammetry and Remote Sensing, 139, 241–254. https://doi.org/10.1016/j.isprsjprs.2018.03.017

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by the KAIST Key Research Institutes Project (Interdisciplinary Research Group) (N11220120).

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology, 291 Daehak-ro, Yuseong-gu, 34141, Daejeon, Republic of Korea

    Jonghwi Kim & Jinwhan Kim

  2. Maritime Technology Research Institute, Agency of Defense Development, Jinhae P.O.Box, Changwon, 51678, Gyeongnam, Republic of Korea

    Yonghoon Cho

Authors
  1. Jonghwi Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yonghoon Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinwhan Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinwhan Kim.

Ethics declarations

Declarations

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, J., Cho, Y. & Kim, J. Urban localization based on aerial imagery by correcting projection distortion. Auton Robot 47, 299–312 (2023). https://doi.org/10.1007/s10514-022-10082-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-022-10082-5

Keywords

Search

Navigation