Skip to main content
SpringerLink
Log in

Correspondence identification for collaborative multi-robot perception under uncertainty

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

Correspondence identification is a critical capability for multi-robot collaborative perception, which allows a group of robots to consistently refer to the same objects in their own fields of view. Correspondence identification is challenging due to the existence of non-covisible objects that cannot be observed by all robots, and due to uncertainty in robot perception. In this paper, we introduce a novel principled approach that formulates correspondence identification as a graph matching problem under the mathematical framework of regularized constrained optimization. We develop a regularization term to explicitly address perception uncertainties by penalizing the object correspondences with a high uncertainty. We also introduce a second regularization term to explicitly address non-covisible objects by penalizing the correspondences built by the non-covisible objects. Our approach is evaluated in robotic simulations and real physical robots. Experimental results show that our method is able to address correspondence identification under uncertainty and non-covisibility, and achieves the state-of-the-art performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. The datasets are available at: http://hcr.mines.edu/project/civr.html.

References

  • Almohamad, H., & Duffuaa, S. O. (1993). A linear programming approach for the weighted graph matching problem. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(5), 522–525.

    Article  Google Scholar 

  • Aragues R, Montijano E, Sagues C (2011) Consistent data association in multi-robot systems with limited communications. In: Robotics: Science and Systems

  • Bertoni L, Kreiss S, Alahi A (2019) MonoLoco: Monocular 3D pedestrian localization and uncertainty estimation. In: IEEE International Conference on Computer Vision

  • Besl, P. J., & McKay, N. D. (1992). Method for registration of 3D shapes. Sensor Fusion, 1611, 586–606.

    Google Scholar 

  • Black, J., & Ellis, T. (2002). Multi-camera image measurement and correspondence. Measurement, 32(1), 61–71.

    Article  Google Scholar 

  • Boroson ER, Ayanian N (2019) 3D Keypoint Repeatability for Heterogeneous Multi-Robot SLAM. In: IEEE International Conference on Robotics and Automation

  • Boyd S, Parikh N, Chu E, Peleato B, Eckstein J, et al. (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning 3(1):1–122

  • Brambilla, M., Ferrante, E., Birattari, M., & Dorigo, M. (2013). Swarm robotics: a review from the swarm engineering perspective. Swarm Intelligence, 7(1), 1–41.

    Article  Google Scholar 

  • Brooks, S., Gelman, A., Jones, G., & Meng, X. L. (2011). Handbook of markov chain monte carlo. Florida: CRC Press.

    Book  Google Scholar 

  • Buschka, P., Saffiotti, A., & Wasik, Z. (2000). Fuzzy landmark-based localization for a legged robot. International Conference on Intelligent Robots and Systems, 2, 1205–1210.

    Google Scholar 

  • Carreras, C., & Walker, I. D. (2001). Interval methods for fault-tree analysis in robotics. IEEE Transactions on Reliability, 50(1), 3–11.

    Article  Google Scholar 

  • Chang, H. J., Fischer, T., Petit, M., Zambelli, M., & Demiris, Y. (2017). Learning kinematic structure correspondences using multi-order similarities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1, 1.

    Google Scholar 

  • Chen Y, Zhu X, Gong S (2017) Person re-identification by deep learning multi-scale representations. In: IEEE International Conference on Computer Vision

  • Cho M, Lee J, Lee KM (2010) Reweighted random walks for graph matching. In: European Conference on Computer Vision

  • Chung, S. J., Paranjape, A. A., Dames, P., Shen, S., & Kumar, V. (2018). A survey on aerial swarm robotics. IEEE Transactions on Robotics, 34(4), 837–855.

    Article  Google Scholar 

  • Der Kiureghian, A., & Ditlevsen, O. (2009). Aleatory or epistemic? Does it matter? Structural Safety, 31(2), 105–112.

    Article  Google Scholar 

  • Ding N, Fang Y, Babbush R, Chen C, Skeel RD, Neven H (2014) Bayesian sampling using stochastic gradient thermostats. In: Advances in neural information processing systems, pp 3203–3211

  • Dogar, M., Spielberg, A., Baker, S., & Rus, D. (2019). Multi-robot grasp planning for sequential assembly operations. Autonomous Robots, 43(3), 649–664.

    Article  Google Scholar 

  • Duchenne, O., Bach, F., Kweon, I. S., & Ponce, J. (2011). A tensor-based algorithm for high-order graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12), 2383–2395.

    Article  Google Scholar 

  • Engel J, Schöps T, Cremers D (2014) LSD-SLAM: Large-scale direct monocular SLAM. In: European Conference on Computer Vision

  • Fathian K, Khosoussi K, Lusk P, Tian Y, How JP (2019) CLEAR: A consistent lifting, rmbedding, and alignment rectification algorithm for multi-agent data association. arXiv

  • Frey KM, Steiner TJ, How JP (2019) Efficient constellation-based map-merging for semantic SLAM. In: IEEE International Conference on Robotics and Automation

  • Gao P, Guo R, Lu H, Zhang H (2020a) Regularized graph matching for correspondence identification under uncertainty in collaborative perception. Robotics: Science and Systems

  • Gao P, Reily B, Paul S, Zhang H (2020b) Visual reference of ambiguous objects for augmented reality-powered human-robot communication in a shared workspace. International Conference on Virtual, Augmented and Mixed Reality

  • Girshick R (2015) Fast R-CNN. In: IEEE international conference on computer vision, pp 1440–1448

  • Gojcic Z, Zhou C, Wegner JD, Wieser A (2019) The perfect match: 3D point cloud matching with smoothed densities. In: IEEE Conference on Computer Vision and Pattern Recognition

  • Gundavarapu NB, Srivastava D, Mitra R, Sharma A, Jain A (2019) Structured Aleatoric Uncertainty in Human Pose Estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, Workshops

  • He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: IEEE international conference on computer vision, pp 2961–2969

  • Hollinger GA, Englot B, Hover F, Mitra U, Sukhatme GS (2012) Uncertainty-driven view planning for underwater inspection. In: IEEE International Conference on Robotics and Automation, pp 4884–4891

  • Hong, H., Yu, H., & Lee, B. H. (2019). Regeneration of normal distributions transform for target lattice based on fusion of truncated Gaussian components. IEEE Robotics and Automation Letters, 4(2), 684–691.

    Article  Google Scholar 

  • Hu J, Zhang Y, Okatani T (2019) Visualization of convolutional neural networks for monocular depth estimation. In: IEEE International Conference on Computer Vision, pp 3869–3878

  • Hu N, Huang Q, Thibert B, Guibas LJ (2018) Distributable consistent multi-object matching. In: IEEE Conference on Computer Vision and Pattern Recognition

  • Kallasi, F., Rizzini, D. L., & Caselli, S. (2016). Fast keypoint features from laser scanner for robot localization and mapping. IEEE Robotics and Automation Letters, 1(1), 176–183.

    Article  Google Scholar 

  • Kendall A, Gal Y (2017) What uncertainties do we need in bayesian deep learning for computer vision? In: Advances in Neural Information Processing Systems

  • Kendall A, Badrinarayanan V, Cipolla R (2015a) Bayesian Segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. arXiv

  • Kendall A, Grimes M, Cipolla R (2015b) PoseNet: A convolutional network for real-time 6-DOF camera relocalization. In: IEEE International Conference on Computer Vision.

  • Kendall A, Gal Y, Cipolla R (2018) Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In: IEEE Conference on Computer Vision and Pattern Recognition

  • Korattikara Balan, A., Rathod, V., Murphy, K. P., & Welling, M. (2015). Bayesian dark knowledge. Advances in Neural Information Processing Systems, 28, 3438–3446.

    Google Scholar 

  • Kraus F, Dietmayer K (2019) Uncertainty estimation in one-stage object detection. In: IEEE Intelligent Transportation Systems Conference, pp 53–60

  • Laina I, Rupprecht C, Belagiannis V, Tombari F, Navab N (2016) Deeper depth prediction with fully convolutional residual networks. In: International conference on 3D vision, IEEE, pp 239–248

  • Lampert, C. H., Nickisch, H., & Harmeling, S. (2014). Attribute-based classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3), 453–465.

    Article  Google Scholar 

  • Learning, M., Rabanser, S., Shchur, O., & Günnemann, S. (2015). Introduction to Tensor Decompositions and their Applications in Machine Learning. Machine Learning, 98(1–2), 1–5.

    MathSciNet  Google Scholar 

  • Lee J, Cho M, Lee KM (2011) Hyper-graph matching via reweighted random walks. In: IEEE Conference on Computer Vision and Pattern Recognition

  • Leonardos S, Zhou X, Daniilidis K (2017) Distributed consistent data association via permutation synchronization. In: IEEE International Conference on Robotics and Automation

  • Leordeanu M, Hebert M (2005) A spectral technique for correspondence problems using pairwise constraints. In: IEEE International Conference on Computer Vision

  • Li, Q., Xiong, R., & Vidal-Calleja, T. (2017). A GMM based uncertainty model for point clouds registration. Robotics and Autonomous Systems, 91, 349–362.

    Article  Google Scholar 

  • Li Y, Gal Y (2017) Dropout inference in Bayesian neural networks with alpha-divergences. In: International Conference on Machine Learning

  • Li Y, Hernández-Lobato JM, Turner RE (2015) Stochastic expectation propagation. Advances in neural information processing systems.

  • Maset E, Arrigoni F, Fusiello A (2017) Practical and efficient multi-view matching. In: IEEE International Conference on Computer Vision

  • Munkres, J. (1957). Algorithms for the assignment and transportation problems. Journal of the Society for Industrial and Applied Mathematics, 5(1), 32–38.

    Article  MathSciNet  Google Scholar 

  • Mur-Artal, R., Montiel, J. M. M., & Tardos, J. D. (2015). ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Transactions on Robotics, 31(5), 1147–1163.

    Article  Google Scholar 

  • Neal RM (2012) Bayesian learning for neural networks, vol 118. Springer Science & Business Media

  • Neiswanger W, Wang C, Xing E (2013) Asymptotically exact, embarrassingly parallel MCMC. arXiv preprint arXiv:1311.4780

  • Neudecker, H. (1969). A note on Kronecker matrix products and matrix equation systems. SIAM Journal on Applied Mathematics, 17(3), 603–606.

    Article  MathSciNet  Google Scholar 

  • Nguyen, A., Ben-Chen, M., Welnicka, K., Ye, Y., & Guibas, L. (2011). An optimization approach to improving collections of shape maps. Computer Graphics Forum, 30, 1481–1491.

    Article  Google Scholar 

  • Nguyen Q, Gautier A, Hein M (2015) A flexible tensor block coordinate ascent scheme for hypergraph matching. In: IEEE Conference on Computer Vision and Pattern Recognition

  • Pachauri D, Kondor R, Singh V (2013) Solving the multi-way matching problem by permutation synchronization. In: Advances in Neural Information Processing Systems

  • Reily B, Reardon C, Zhang H (2020) Representing Multi-Robot Structure through Multimodal Graph Embedding for the Selection of Robot Teams. arXiv preprint arXiv:2003.12164

  • Richard, M. D., & Lippmann, R. P. (1991). Neural network classifiers estimate bayesian a posterior probabilities. Neural computation, 3(4), 461–483.

    Article  Google Scholar 

  • Robin, C., & Lacroix, S. (2016). Multi-robot target detection and tracking: taxonomy and survey. Autonomous Robots, 40(4), 729–760.

    Article  Google Scholar 

  • Senanayake, M., Senthooran, I., Barca, J. C., Chung, H., Kamruzzaman, J., & Murshed, M. (2016). Search and tracking algorithms for swarms of robots: A survey. Robotics and Autonomous Systems, 75, 422–434.

    Article  Google Scholar 

  • Sobreira, H., Costa, C. M., Sousa, I., Rocha, L., Lima, J., Farias, P., et al. (2019). Map-matching algorithms for robot self-localization: a comparison between perfect match, iterative closest point and normal distributions transform. Journal of Intelligent & Robotic Systems, 93(3–4), 533–546.

    Article  Google Scholar 

  • Suh Y, Adamczewski K, Mu Lee K (2015) Subgraph matching using compactness prior for robust feature correspondence. In: IEEE Conference on Computer Vision and Pattern Recognition

  • Szeliski R (2010) Computer vision: Algorithms and applications. Springer Science & Business Media

  • Thrun, S. (2002). Probabilistic robotics. Communications of the ACM, 45(3), 52–57.

    Article  Google Scholar 

  • Tian Y, Liu K, Ok K, Tran L, Allen D, Roy N, How JP (2019) Search and rescue under the forest canopy using multiple UAVs. arXiv

  • Tron R, Zhou X, Esteves C, Daniilidis K (2017) Fast multi-image matching via density-based clustering. In: IEEE International Conference on Computer Vision

  • Wang R, Yan J, Yang X (2019) Learning combinatorial embedding networks for deep graph matching. In: IEEE International Conference on Computer Vision

  • Wei, S., Yu, D., Guo, C. L., Dan, L., & Shu, W. W. (2018). Survey of connected automated vehicle perception mode: from autonomy to interaction. IET Intelligent Transport Systems, 13(3), 495–505.

    Article  Google Scholar 

  • Yan J, Ren Z, Zha H, Chu S (2016) A constrained clustering based approach for matching a collection of feature sets. In: International Conference on Pattern Recognition

  • Yan, Z., Jouandeau, N., & Cherif, A. A. (2013). A survey and analysis of multi-robot coordination. International Journal of Advanced Robotic System, 10(12), 399.

    Article  Google Scholar 

  • Zhang Q, Pless R (2004) Extrinsic calibration of a camera and laser range finder (improves camera calibration). In: IEEE/RSJ International Conference on Intelligent Robots and Systems

  • Zhang R, Wang W (2016) An MCMC-based prior sub-hypergraph matching in presence of outliers. In: International Conference on Pattern Recognition

  • Zhao, R., Oyang, W., & Wang, X. (2016). Person re-identification by saliency learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(2), 356–370.

    Article  Google Scholar 

  • Zhao Y, Shen X, Jin Z, Lu H, Hua Xs (2019) Attribute-driven feature disentangling and temporal aggregation for video person re-identification. In: IEEE Conference on Computer Vision and Pattern Recognition

  • Zhou X, Zhu M, Daniilidis K (2015) Multi-image matching via fast alternating minimization. In: IEEE International Conference on Computer Vision

Download references

Acknowledgements

This work was partially supported by the NSF CAREER award IIS-1942056, DARPA Young Faculty Award (YFA) D21AP10114-00, and ARL DCIST CRA W911NF-17-2-0181.

Author information

Authors and Affiliations

  1. Colorado School of Mines, 1500 Illinois St, Golden, CO, 80401, USA

    Peng Gao & Hao Zhang

  2. Toyota Motor North America, R&D InfoTech Labs, 465 N Bernardo Ave, Mountain View, CA, 94043, USA

    Rui Guo & Hongsheng Lu

Authors
  1. Peng Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rui Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongsheng Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peng Gao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This is one of the several papers published in Autonomous Robots comprising the Special Issue on Robotics: Science and Systems 2020.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, P., Guo, R., Lu, H. et al. Correspondence identification for collaborative multi-robot perception under uncertainty. Auton Robot 46, 5–20 (2022). https://doi.org/10.1007/s10514-021-10009-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-021-10009-6

Keywords

Search

Navigation