skip to main content
research-article

Asynchronous Collaborative Autoscanning with Mode Switching for Multi-Robot Scene Reconstruction

Published: 30 November 2022 Publication History

Abstract

When conducting autonomous scanning for the online reconstruction of unknown indoor environments, robots have to be competent at exploring scene structure and reconstructing objects with high quality. Our key observation is that different tasks demand specialized scanning properties of robots: rapid moving speed and far vision for global exploration and slow moving speed and narrow vision for local object reconstruction, which are referred as two different scanning modes: explorer and reconstructor, respectively. When requiring multiple robots to collaborate for efficient exploration and fine-grained reconstruction, the questions on when to generate and how to assign those tasks should be carefully answered. Therefore, we propose a novel asynchronous collaborative autoscanning method with mode switching, which generates two kinds of scanning tasks with associated scanning modes, i.e., exploration task with explorer mode and reconstruction task with reconstructor mode, and assign them to the robots to execute in an asynchronous collaborative manner to highly boost the scanning efficiency and reconstruction quality. The task assignment is optimized by solving a modified Multi-Depot Multiple Traveling Salesman Problem (MDMTSP). Moreover, to further enhance the collaboration and increase the efficiency, we propose a task-flow model that actives the task generation and assignment process immediately when any of the robots finish all its tasks with no need to wait for all other robots to complete the tasks assigned in the previous iteration. Extensive experiments have been conducted to show the importance of each key component of our method and the superiority over previous methods in scanning efficiency and reconstruction quality.

Supplemental Material

MP4 File
presentation

References

[1]
Kamalova Albina and Suk Gyu Lee. 2019. Hybrid stochastic exploration using grey wolf optimizer and coordinated multi-robot exploration algorithms. IEEE Access 7 (2019), 14246--14255.
[2]
Robin Amsters and Peter Slaets. 2019. Turtlebot 3 as a robotics education platform. In International Conference on Robotics in Education (RiE). Springer, 170--181.
[3]
Nikolay Atanasov, Jerome Le Ny, Kostas Daniilidis, and George J Pappas. 2015. Decentralized active information acquisition: Theory and application to multi-robot SLAM. In 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 4775--4782.
[4]
Subhrajit Bhattacharya, Robert Ghrist, and Vijay Kumar. 2014. Multi-robot coverage and exploration on Riemannian manifolds with boundaries. The International Journal of Robotics Research 33, 1 (2014), 113--137.
[5]
Wolfram Burgard, Mark Moors, Cyrill Stachniss, and Frank E Schneider. 2005. Coordinated multi-robot exploration. IEEE Transactions on robotics 21, 3 (2005), 376--386.
[6]
John Canny. 1986. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence 6 (1986), 679--698.
[7]
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158 (2017).
[8]
Benjamin Charrow, Gregory Kahn, Sachin Patil, Sikang Liu, Ken Goldberg, Pieter Abbeel, Nathan Michael, and Vijay Kumar. 2015. Information-Theoretic Planning with Trajectory Optimization for Dense 3D Mapping. In Robotics: Science and Systems, Vol. 11. Rome, 3--12.
[9]
Omar Cheikhrouhou and Ines Khoufi. 2021. A comprehensive survey on the Multiple Traveling Salesman Problem: Applications, approaches and taxonomy. Computer Science Review 40 (2021), 100369.
[10]
Xinye Chen, Ping Zhang, Guanglong Du, and Fang Li. 2018. Ant colony optimization based memetic algorithm to solve bi-objective multiple traveling salesmen problem for multi-robot systems. IEEE Access 6 (2018), 21745--21757.
[11]
Siyan Dong, Kai Xu, Qiang Zhou, Andrea Tagliasacchi, Shiqing Xin, Matthias Nießner, and Baoquan Chen. 2019. Multi-robot collaborative dense scene reconstruction. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--16.
[12]
Thibaud Duhautbout, Julien Moras, and Julien Marzat. 2019. Distributed 3D TSDF manifold mapping for multi-robot systems. In 2019 European Conference on Mobile Robots (ECMR). IEEE, 1--8.
[13]
Jan Faigl, Miroslav Kulich, and Libor Přeučil. 2012. Goal assignment using distance cost in multi-robot exploration. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 3741--3746.
[14]
Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. 2019. Scene memory transformer for embodied agents in long-horizon tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 538--547.
[15]
Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 2021. 3d-future: 3d furniture shape with texture. International Journal of Computer Vision (2021), 1--25.
[16]
M. Grinvald, F. Furrer, T. Novkovic, J. J. Chung, C. Cadena, R. Siegwart, and J. Nieto. 2019. Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery. IEEE Robotics and Automation Letters 4, 3 (July 2019), 3037--3044.
[17]
Li Guan. 2006. Sensor-based cooperative multi-robot 3d environment reconstruction. (2006).
[18]
Ankur Handa, Thomas Whelan, John McDonald, and Andrew J Davison. 2014. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In 2014 IEEE international conference on Robotics and automation (ICRA). IEEE, 1524--1531.
[19]
Peter E Hart, Nils J Nilsson, and Bertram Raphael. 1968. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics 4, 2 (1968), 100--107.
[20]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. 2961--2969.
[21]
Lionel Heng, Alkis Gotovos, Andreas Krause, and Marc Pollefeys. 2015. Efficient visual exploration and coverage with a micro aerial vehicle in unknown environments. In 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 1071--1078.
[22]
Junyan Hu, Hanlin Niu, Joaquin Carrasco, Barry Lennox, and Farshad Arvin. 2020. Voronoi-based multi-robot autonomous exploration in unknown environments via deep reinforcement learning. IEEE Transactions on Vehicular Technology 69, 12 (2020), 14413--14423.
[23]
Pengdi Huang, Liqiang Lin, Kai Xu, and Hui Huang. 2020. Autonomous Outdoor Scanning via Online Topological and Geometric Path Optimization. IEEE Transactions on Intelligent Transportation Systems (2020).
[24]
Miguel Juliá, Arturo Gil, and Oscar Reinoso. 2012. A comparison of path planning strategies for autonomous exploration and mapping of unknown environments. Autonomous Robots 33, 4 (2012), 427--444.
[25]
Ines Khoufi, Mohamed Hadded, Pascale Minet, and Anis Laouiti. 2015. Optimized trajectories of multi-robot deploying wireless sensor nodes. In 2015 IEEE 11th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob). IEEE, 724--731.
[26]
Nathan Koenig and Andrew Howard. 2004. Design and use paradigms for gazebo, an open-source multi-robot simulator. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566), Vol. 3. IEEE, 2149--2154.
[27]
Michael Krainin, Brian Curless, and Dieter Fox. 2011. Autonomous generation of complete 3D object models using next best view manipulation planning. In 2011 IEEE International Conference on Robotics and Automation. IEEE, 5031--5037.
[28]
Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. 2019. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 4213--4220.
[29]
Ligang Liu, Xi Xia, Han Sun, Qi Shen, Juzhan Xu, Bin Chen, Hui Huang, and Kai Xu. 2018. Object-aware guidance for autonomous scene reconstruction. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--12.
[30]
Yilin Liu, Ruiqi Cui, Ke Xie, Minglun Gong, and Hui Huang. 2021. Aerial Path Planning for Online Real-Time Exploration and Offline High-Quality Reconstruction of Large-Scale Urban Scenes. ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA) 40, 6 (2021), 226:1--226:16.
[31]
Fethi Matoui, Boumedyen Boussaid, Brahim Metoui, and Mohamed Naceur Abdelkrim. 2020. Contribution to the path planning of a multi-robot system: centralized architecture. Intelligent Service Robotics 13, 1 (2020), 147--158.
[32]
Farzad Niroui, Kaicheng Zhang, Zendai Kashino, and Goldie Nejat. 2019. Deep reinforcement learning robot for search and rescue applications: Exploration in unknown cluttered environments. IEEE Robotics and Automation Letters 4, 2 (2019), 610--617.
[33]
Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, Andrew Y Ng, et al. 2009. ROS: an open-source Robot Operating System. In ICRA workshop on open source software, Vol. 3. Kobe, Japan, 5.
[34]
Manikandasriram Srinivasan Ramanagopal and Jerome Le Ny. 2016. Motion planning strategies for autonomously mapping 3d structures. arXiv preprint arXiv:1602.06667 (2016).
[35]
Mike Roberts, Debadeepta Dey, Anh Truong, Sudipta Sinha, Shital Shah, Ashish Kapoor, Pat Hanrahan, and Neel Joshi. 2017. Submodular trajectory optimization for aerial 3d scanning. In Proceedings of the IEEE International Conference on Computer Vision. 5324--5333.
[36]
Guillaume Sartoretti, Yue Wu, William Paivine, TK Satish Kumar, Sven Koenig, and Howie Choset. 2019. Distributed reinforcement learning for multi-robot decentralized collective construction. In Distributed autonomous robotic systems. Springer, 35--49.
[37]
Lukas Schmid, Michael Pantic, Raghav Khanna, Lionel Ott, Roland Siegwart, and Juan Nieto. 2020. An efficient sampling-based method for online informative path planning in unknown environments. IEEE Robotics and Automation Letters 5, 2 (2020), 1500--1507.
[38]
Miguel Schneider-Fontan and Maja J Mataric. 1998. Territorial multi-robot task division. IEEE Transactions on Robotics and Automation 14, 5 (1998), 815--822.
[39]
Maks Sorokin, Wenhao Yu, Sehoon Ha, and C Karen Liu. 2021. Learning human search behavior from egocentric visual inputs. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 389--398.
[40]
Kaarthik Sundar and Sivakumar Rathinam. 2017. Algorithms for heterogeneous, multiple depot, multiple unmanned vehicle path planning problems. Journal of Intelligent & Robotic Systems 88, 2 (2017), 513--526.
[41]
Yulun Tian, Yun Chang, Fernando Herrera Arias, Carlos Nieto-Granda, Jonathan P How, and Luca Carlone. 2021. Kimera-multi: Robust, distributed, dense metric-semantic slam for multi-robot systems. arXiv preprint arXiv:2106.14386 (2021).
[42]
Sahar Trigui, Omar Cheikhrouhou, Anis Koubaa, Uthman Baroudi, and Habib Youssef. 2017. FL-MTSP: a fuzzy logic approach to solve the multi-objective multiple traveling salesman problem for multi-robot systems. Soft Computing 21, 24 (2017), 7351--7362.
[43]
Masoumeh Vali and Khodakaram Salimifard. 2017. A constraint programming approach for solving multiple traveling salesman problem. In The Sixteenth International Workshop on Constraint Modelling and Reformulation. 1--17.
[44]
J Irving Vasquez-Gomez, L Enrique Sucar, Rafael Murrieta-Cid, and Efrain Lopez-Damian. 2014. Volumetric next-best-view planning for 3D object reconstruction with positioning error. International Journal of Advanced Robotic Systems 11, 10 (2014), 159.
[45]
Melonee Wise, Michael Ferguson, Derek King, Eric Diehr, and David Dymesich. 2016. Fetch and freight: Standard platforms for service robot applications. In Workshop on autonomous mobile service robots.
[46]
Shihao Wu, Wei Sun, Pinxin Long, Hui Huang, Daniel Cohen-Or, Minglun Gong, Oliver Deussen, and Baoquan Chen. 2014. Quality-driven poisson-guided autoscanning. ACM Transactions on Graphics 33, 6 (2014).
[47]
Haozhe Xie, Hongxun Yao, Shangchen Zhou, Jiageng Mao, Shengping Zhang, and Wenxiu Sun. 2020. Grnet: Gridding residual network for dense point cloud completion. In European Conference on Computer Vision. Springer, 365--381.
[48]
Kai Xu, Hui Huang, Yifei Shi, Hao Li, Pinxin Long, Jianong Caichen, Wei Sun, and Baoquan Chen. 2015. Autoscanning for coupled scene reconstruction and proactive object analysis. ACM Transactions on Graphics (TOG) 34, 6 (2015), 1--14.
[49]
Kai Xu, Yifei Shi, Lintao Zheng, Junyu Zhang, Min Liu, Hui Huang, Hao Su, Daniel Cohen-Or, and Baoquan Chen. 2016. 3D attention-driven depth acquisition for object identification. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1--14.
[50]
Kai Xu, Lintao Zheng, Zihao Yan, Guohang Yan, Eugene Zhang, Matthias Niessner, Oliver Deussen, Daniel Cohen-Or, and Hui Huang. 2017. Autonomous reconstruction of unknown indoor scenes guided by time-varying tensor fields. ACM Transactions on Graphics (TOG) 36, 6 (2017), 1--15.
[51]
Sai Yadlapalli, Waqar A Malik, Swaroop Darbha, and Meir Pachter. 2009. A Lagrangian-based algorithm for a multiple depot, multiple traveling salesmen problem. Nonlinear Analysis: Real World Applications 10, 4 (2009), 1990--1999.
[52]
Kai Ye, Siyan Dong, Qingnan Fan, He Wang, Li Yi, Fei Xia, Jue Wang, and Baoquan Chen. 2022. Multi-Robot Active Mapping via Neural Bipartite Graph Matching. arXiv preprint arXiv:2203.16319 (2022).
[53]
Shuai Yuan, Bradley Skinner, Shoudong Huang, and Dikai Liu. 2013. A new crossover approach for solving the multiple travelling salesmen problem using genetic algorithms. European journal of operational research 228, 1 (2013), 72--82.

Cited By

View all
  • (2025)Point cloud quality metrics for incremental image-based 3D reconstructionMultimedia Tools and Applications10.1007/s11042-025-20596-6Online publication date: 8-Jan-2025
  • (2024)LightFormer: Light-Oriented Global Neural Rendering in Dynamic SceneACM Transactions on Graphics10.1145/365822943:4(1-14)Online publication date: 19-Jul-2024
  • (2024)Autonomous Implicit Indoor Scene Reconstruction with Frontier Exploration2024 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA57147.2024.10611382(18041-18047)Online publication date: 13-May-2024
  • Show More Cited By

Index Terms

  1. Asynchronous Collaborative Autoscanning with Mode Switching for Multi-Robot Scene Reconstruction

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Graphics
    ACM Transactions on Graphics  Volume 41, Issue 6
    December 2022
    1428 pages
    ISSN:0730-0301
    EISSN:1557-7368
    DOI:10.1145/3550454
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 30 November 2022
    Published in TOG Volume 41, Issue 6

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. asynchronous task assignment
    2. autonomous reconstruction
    3. indoor scene reconstruction
    4. multiple robots cooperation

    Qualifiers

    • Research-article

    Funding Sources

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)53
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Point cloud quality metrics for incremental image-based 3D reconstructionMultimedia Tools and Applications10.1007/s11042-025-20596-6Online publication date: 8-Jan-2025
    • (2024)LightFormer: Light-Oriented Global Neural Rendering in Dynamic SceneACM Transactions on Graphics10.1145/365822943:4(1-14)Online publication date: 19-Jul-2024
    • (2024)Autonomous Implicit Indoor Scene Reconstruction with Frontier Exploration2024 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA57147.2024.10611382(18041-18047)Online publication date: 13-May-2024
    • (2024)GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01555(16436-16445)Online publication date: 16-Jun-2024
    • (2024)Emerging Trends in Realistic Robotic Simulations: A Comprehensive Systematic Literature ReviewIEEE Access10.1109/ACCESS.2024.3404881(1-1)Online publication date: 2024
    • (2023)Online Scene CAD Recomposition via Autonomous ScanningACM Transactions on Graphics10.1145/361833942:6(1-16)Online publication date: 5-Dec-2023
    • (2023)Surface and Edge Detection for Primitive Fitting of Point CloudsACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591522(1-10)Online publication date: 23-Jul-2023
    • (2023)PPI-NET: End-to-End Parametric Primitive InferenceAdvances in Computer Graphics10.1007/978-3-031-50078-7_6(67-78)Online publication date: 28-Aug-2023
    • (2022)Reconstructing editable prismatic CAD from rounded voxel modelsSIGGRAPH Asia 2022 Conference Papers10.1145/3550469.3555424(1-9)Online publication date: 29-Nov-2022

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media