Abstract
This paper presents a cross-domain and cross-view framework for underwater robot localisation, which does not require any Global Positioning System (GPS) information. The proposed localisation method uses colour aerial images and underwater acoustic images to estimate the robot’s position. The method identifies the correlation among images from distinct domains, given by the matching of images acquired in partially structured environments with shared features. The validation of the proposed method is done using a real dataset, which was acquired by an underwater vehicle in a Marina. Besides, it was compared to Dead Reckoning and a learning-based particle filter method. The experimental results present the feasibility of the proposed method and its advances in relation to state-of-the-art algorithms.
Similar content being viewed by others
Data Availability
The code that support the findings of this study are available from the corresponding author upon reasonable request.
References
Concha, A., Drews-Jr, P., Campos, M., Civera, J.: Real-time localization and dense mapping in underwater environments from a monocular sequence. In: OCEANS 2015 - Genova, pp. 1–5 (2015). 10.1109/OCEANS-Genova.2015.7271476
Kuhn, V.N., Drews-Jr, P.L.J., Gomes, S.C.P., Cunha, M.A.B., Botelho, S.S.d.C.: Automatic control of a rov for inspection of underwater structures using a low-cost sensing. Journal of the Brazilian Society of Mechanical Sciences and Engineering 37(1), 361–374 (2015)
Petillot, Y.R., Antonelli, G., Casalino, G., Ferreira, F.: Underwater robots: From remotely operated vehicles to intervention-autonomous underwater vehicles. IEEE RA Magazine 26(2), 94–101 (2019). https://doi.org/10.1109/MRA.2019.2908063
Siddall, R., Kovač, M.: Launching the aquamav: bioinspired design for aerial-aquatic robotic platforms. Bioinspiration & Biomimetics 9(3), 031001 (2014)
Sathappan, N., Tokhi, M.O., Penaluna, L., Zhao, Z., Duan, F., Shirkoohi, G., Kaur, A.: A literature review on data transmission using electromagnetic waves under different aquatic environments. Marine Technology Society Journal 55(5), 138–149 (2021)
Drews-Jr, P., Nascimento, E.R., Campos, M.F.M., Elfes, A.: Automatic restoration of underwater monocular sequences of images. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1058–1064 (2015). https://doi.org/10.1109/IROS.2015.7353501
Paull, L., Saeedi, S., Seto, M., Li, H.: Auv navigation and localization: A review. IEEE J. Oceanic Eng. 39(1), 131–149 (2013)
C. S. Ribeiro, P.O., M. dos Santos, M., L. J. Drews-Jr, P., S. C. Botelho, S.: Forward looking sonar scene matching using deep learning. In: 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 574–579 (2017). https://doi.org/10.1109/ICMLA.2017.00-99
Remmas, W., Chemori, A., Kruusmaa, M.: Diver tracking in open waters: A low-cost approach based on visual and acoustic sensor fusion. J. Field Robotics 38(3), 494–508 (2021)
Hurtos, N., Nagappa, S., Cufi, X., Petillot, Y., Salvi, J.: Evaluation of registration methods on two-dimensional forward-looking sonar imagery. In: OCEANS - Bergen, 2013 MTS/IEEE, pp. 1–8 (2013). https://doi.org/10.1109/OCEANS-Bergen.2013.6608124
Wu, Y.: Coordinated path planning for an unmanned aerial-aquatic vehicle (UAAV) and an autonomous underwater vehicle (AUV) in an underwater target strike mission. Ocean Engineering 182, 162–173 (2019). https://doi.org/10.1016/j.oceaneng.2019.04.062
Leung, K.Y.K., Clark, C.M., Huissoon, J.P.: Localization in urban environments by matching ground level video images with an aerial image. In: IEEE ICRA 2008, pp. 551–556 (2008). https://doi.org/10.1109/ROBOT.2008.4543264
Viswanathan, A., Pires, B.R., Huber, D.: Vision based robot localization by ground to satellite matching in gps-denied situations. In: 2014 IEEE/RSJ IROS, pp. 192–198 (2014). https://doi.org/10.1109/IROS.2014.6942560
Wolff, M., Collins, R.T., Liu, Y.: Regularity-driven building facade matching between aerial and street views. In: IEEE CVPR, pp. 1591–1600 (2016). https://doi.org/10.1109/CVPR.2016.176
Hu, S., Feng, M., Nguyen, R.M.H., Hee Lee, G.: CVM-Net: Cross-view matching network for image-based ground-to-aerial geo-localization. In: IEEE CVPR, pp. 7258–7267 (2018)
Zhuo, X., Koch, T., Kurz, F., Fraundorfer, F., Reinartz, P.: Automatic uav image geo-registration by matching uav images to georeferenced image data. Remote Sensing 9(4), 376 (2017)
Drews-Jr, P., Neto, A., Campos, M.: Hybrid unmanned aerial underwater vehicle: Modeling and simulation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4637–4642 (2014). https://doi.org/10.1109/IROS.2014.6943220
Horn, A.C., Pinheiro, P.M., Grando, R.B., da Silva, C.B., Neto, A.A., Drews-Jr, P.L.J.: A novel concept for hybrid unmanned aerial underwater vehicles focused on aquatic performance. In: IEEE Latin American Robotics Symposium (LARS) and Brazilian Symposium on Robotics (SBR), pp. 1–6 (2020). https://doi.org/10.1109/LARS/SBR/WRE51543.2020.9307110
Dos Santos, M.M., De Giacomo, G.G., Drews-Jr, P.L.J., Botelho, S.S.C.: Satellite and underwater sonar image matching using deep learning. In: 2019 Latin American Robotics Symposium (LARS), 2019 Brazilian Symposium on Robotics (SBR) and 2019 Workshop on Robotics in Education (WRE), pp. 109–114 (2019). https://doi.org/10.1109/LARS-SBR-WRE48964.2019.00027
Dos Santos, M.M., De Giacomo, G.G., Drews, P.L.J., Botelho, S.S.C.: Matching color aerial images and underwater sonar images using deep learning for underwater localization. IEEE Robotics and Automation Letters 5(4), 6365–6370 (2020)
Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: IEEE CVPR, vol. 1, pp. 539–546 (2005)
De Giacomo, G.G., dos Santos, M.M., Drews-Jr, P.L., Botelho, S.S.: Cooperative training of triplet networks for cross-domain matching. In: 2020 Latin American Robotics Symposium (LARS), 2020 Brazilian Symposium on Robotics (SBR) and 2020 Workshop on Robotics in Education (WRE), pp. 1–6 (2020)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: International Workshop on Similarity-Based Pattern Recognition, pp. 84–92 (2015). Springer
Thrun, S.: Probabilistic robotics. Communications of the ACM 45(3), 52–57 (2002)
Dos Santos, M.M., De Giacomo, G.G., Drews-Jr, P.L., Botelho, S.S., Mello, C.D.: A framework for underwater vehicle localization based on cross-view and cross-domain acoustic and aerial images. In: 2021 Latin American Robotics Symposium (LARS), 2021 Brazilian Symposium on Robotics (SBR), and 2021 Workshop on Robotics in Education (WRE), pp. 204–209 (2021). IEEE
Dos Santos, M.M., De Giacomo, G.G., Drews-Jr, P.L., Botelho, S.S.: Cross-view and cross-domain underwater localization based on optical aerial and acoustic underwater images. IEEE Robotics and Automation Letters 7(2), 4969–4974 (2022)
Djuric, P.M., Kotecha, J.H., Zhang, J., Huang, Y., Ghirmai, T., Bugallo, M.F., Miguez, J.: Particle filtering. IEEE Signal Processing Magazine 20(5), 19–38 (2003)
Gao, X., Shen, S., Hu, Z., Wang, Z.: Ground and aerial meta-data integration for localization and reconstruction: A review. Pattern Recognition Letters 127, 202–214 (2018)
Bansal, M., Daniilidis, K., Sawhney, H.: Ultrawide baseline facade matching for geo-localization. In: Large-Scale Visual Geo-Localization, pp. 77–98. Springer, (2016)
Bansal, M., Sawhney, H.S., Cheng, H., Daniilidis, K.: Geo-localization of street views with aerial image databases. In: Proceedings of the 19th ACM International Conference on Multimedia. MM ’11, pp. 1125–1128. ACM, New York, NY, USA (2011). 10.1145/2072298.2071954
Li, A., Morariu, V.I., Davis, L.S.: Planar structure matching under projective uncertainty for geolocation. In: ECCV, pp. 265–280 (2014)
Majdik, A.L., Albers-Schoenberg, Y., Scaramuzza, D.: MAV urban localization from google street view data. In: IEEE/RSJ IROS, pp. 3979–3986 (2013)
Noda, M., Takahashi, T., Deguchi, D., Ide, I., Murase, H., Kojima, Y., Naito, T.: Vehicle ego-localization by matching in-vehicle camera images to an aerial image. In: ACCV, pp. 163–173 (2010)
Lin, T.-Y., Belongie, S., Hays, J.: Cross-view image geolocalization. In: IEEE CVPR, pp. 891–898 (2013)
Castaldo, F., Zamir, A., Angst, R., Palmieri, F., Savarese, S.: Semantic cross-view matching. In: IEEE ICCVw, pp. 9–17 (2015)
Lin, T.-Y., Cui, Y., Belongie, S., Hays, J.: Learning deep representations for ground-to-aerial geolocalization. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
Workman, S., Jacobs, N.: On the location dependence of convolutional neural network features. In: IEEE CVPRw, pp. 70–78 (2015)
Workman, S., Souvenir, R., Jacobs, N.: Wide-area image geolocalization with aerial reference imagery. In: The IEEE International Conference on Computer Vision (ICCV) (2015)
Tian, Y., Chen, C., Shah, M.: Cross-view image matching for geo-localization in urban environments. In: IEEE CVPR, pp. 3608–3616 (2017)
Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 28. Curran Associates, Inc., (2015). https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf
Matas, J., Galambos, C., Kittler, J.: Robust detection of lines using the progressive probabilistic hough transform. CVIU 78(1), 119–137 (2000)
Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: Speeded-up robust features (surf). CVIU 110(3), 346–359 (2008). https://doi.org/10.1016/j.cviu.2007.09.014
Drews-Jr, P., Longui, J., Rosa, V.: Real-time depth estimation for underwater inspection using dual laser and camera. In: Symposium on Computing and Automation for Offshore Shipbuilding (NAVCOMP), pp. 52–56 (2013). https://doi.org/10.1109/NAVCOMP.2013.16
Armstrong, R.A., Pizarro, O., Roman, C.: Underwater robotic technology for imaging mesophotic coral ecosystems. In: Mesophotic Coral Ecosystems, pp. 973–988. Springer, (2019)
Greenaway, S.F., Sullivan, K.D., Umfress, S.H., Beittel, A.B., Wagner, K.D.: Revised depth of the challenger deep from submersible transects; including a general method for precise, pressure-derived depths in the ocean. Deep Sea Research Part I: Oceanographic Research Papers 178, 103644 (2021)
dos Santos, M.M., De Giacomo, G.G., Drews, P.L.J., Botelho, S.S.C.: Semantic segmentation of static and dynamic structures of marina satellite images using deep learning. In: 2019 8th Brazilian Conference on Intelligent Systems (BRACIS), pp. 711–716 (2019). https://doi.org/10.1109/BRACIS.2019.00129
Irwin, F., et al.: An isotropic 3x3 image gradient operator. Presentation at Stanford AI Project 2014(02) (1968)
Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: Ros: an open-source robot operating system. In: ICRA Workshop on Open Source Software, vol. 3, p. 5 (2009). Kobe, Japan
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI, pp. 234–241 (2015)
Acknowledgements
This research is partly supported by CNPq, CAPES and FAPERGS. We also would like to thank the colleagues from NAUTEC-FURG for helping with the experimental data and for productive discussions and meetings. Finally, we would like to thank NVIDIA for donating high-performance graphics cards. All authors are with NAUTEC, Intelligent Robotics and Automation Group, Federal University of Rio Grande - FURG, Rio Grande - Brazil.
Author information
Authors and Affiliations
Contributions
Matheus M. dos Santos: Conceptualisation; Methodology; Software; Validation; Investigation; Writing - Original Draft.Paulo J. D. O. Evald: Writing - Review & Editing. Giovanni G. De Giacomo: Data Curation; Formal analysis. Paulo L. J. Drews-Jr: Visualisation; Project administration; Review & Editing. Silvia S. C. Botelho: Supervision; Resources.
Corresponding author
Ethics declarations
Ethics Approval
Not applicable.
Consent to Participate
Not applicable.
Consent to Publish
Not applicable.
Conflicts of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
dos Santos, M.M., de Oliveira Evald, P.J.D., de Giacomo, G.G. et al. A Probabilistic Underwater Localisation based on Cross-view and Cross-domain Acoustic and Aerial Images. J Intell Robot Syst 108, 34 (2023). https://doi.org/10.1007/s10846-023-01837-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-023-01837-y