A high efficient multi-robot simultaneous localization and mapping system using partial computing offloading assisted cloud point registration strategy

https://doi.org/10.1016/j.jpdc.2020.10.012Get rights and content

Highlights

  • Proposed a novel task offloading strategy combined with a new dense point cloud map construction method.

  • Developed a novel strategy to remotely offload computation-intensive tasks to cloud center.

  • Proposed a modified iterative closest point algorithm (ICP), named fitness score hierarchical ICP algorithm (FS-HICP), to accelerate point cloud registration.

Abstract

The robots using visual simultaneous localization and mapping (SLAM) system are generally experiencing excessive power consumption and suffer from depletion of battery energy during the course of working. The intensive computation necessary to complete complicated tasks is overwhelming for inexpensive mobile robots with limited on-board resources. To address this problem, a novel task offloading strategy combined with a new dense point cloud map construction method is proposed in this paper, which is firstly used for the improvement of the system especially in indoor scenes. First, we develop a novel strategy to remotely offload computation-intensive tasks to cloud center so that the tasks that could not originally be achieved locally on the resource-limited robot systems become possible. Second, a modified iterative closest point algorithm (ICP), named fitness score hierarchical ICP algorithm (FS-HICP), is developed to accelerate point cloud registration. The correctness, efficiency, and scalability of the proposed strategy are evaluated with both theoretical analysis and experimental simulations. The results show that the proposed method can effectively reduce the energy consumption while increase the computation capability and speed of the multi-robot visual SLAM system, especially in indoor environment.

Introduction

The aim of simultaneous localization and mapping (SLAM) technology is to provide good estimation of both the robot pose and the map in unknown environments, where an accurate map is unavailable [16]. With the rapid development of onboard hardware, software and machine learning technology, increasing research interests have been witnessed on using SLAM technology to solve various complicated tasks in daily life, e.g., fire rescue, geological exploration, underwater search and archeological excavation. In order to adapt to practical engineering applications, one of the most notable research interests is three-dimensional SLAM. At present laser radar assisted camera is always used as signal inputs in 3D-SLAM solutions and commercially available products, such as drones. However, due to the high cost, fast power consumption and low precision of laser radar, the application of the laser radar based 3D-SLAM technique is hindered to be widely used in the future.

3D visual SLAM, which uses depth cameras as the only sensor inputs, is considered to be beneficial in scenarios where the requirement of cost, energy and weight of the system is strict. Due to various advanced cameras, 3D visual SLAM has attracted more and more attention and opens up a whole new range of possibilities in robot autonomous navigation field, e.g., virtual reality (VR), augmented reality (AR), and of course, autonomous driving technology. However, 3D visual SLAM still faces the challenges of extensive amount of computation and communication tasks where powerful CPU and other on-board resources are required. It turns out that the cost of a single robot is too high to be applied to the actual scenario. Thus, a new performing model or strategy is of great necessity in the 3D visual SLAM process.

To overcome the aforementioned problems, more research works have been conducted to incorporate mobile multi-robot system into SLAM, so that the complexity of a task can be shared by a group of small, often less expensive mobile robots. Generally speaking, mobile robots are limited in size and power, which hinders them from carrying powerful computation and storage unit. Consequently, it might not be fair to ask them to perform extensive computation locally. In order to get rid of this bondage, the “cloud robotics” proposed by Dr. Kuffner [13] in 2010 provides us with a new way to handle complex robot tasks. In cooperation with the cloud platform and data center, robotic network can improve its ability by a large margin. As for applying cloud robotics to solve multi-robot visual SLAM problems, there are still some challenges to be tackled.

We summarize the main problems faced by the multi-robot 3D visual SLAM into the following three aspects.

  • (1)

    The main task of 3D visual SLAM system is image processing, which requires intensive calculation. However, for most inexpensive robotic embedded devices (e.g., Raspberry Pi), it can hardly be handled in real time  [5], [35], [37]. (Also, some recent hardware like NVIDIA Jetson Nano provides us with powerful GPU to process the images. But the price of NVIDIA Jetson Nano is about two times of Raspberry Pi with similar hardware [1]. On the other hand, the battery consumption is always proportional to the circuit complexity. The strategy proposed in the paper is also expected to be applicable in such embedded devices.) Moreover, some high robustness visual SLAM systems, such as ORB-SLAM [25] and RTAP-MAP [15], can only be calculated by powerful computer or even a server. Therefore, how to apply these visual SLAM systems to robotic network needs to be investigated.

  • (2)

    In order to solve the problem of excessive calculations, some researchers have proposed using cloud robotics to offload tasks to cloud. However, images are the only data input for visual SLAM system. The memory space occupied by images is very large, which may cause certain problems while transmitting, such as communication loss and unexpected latency. For example, Wi-Fi connections are not invariably usable and reliable [29]. Moreover, security issues such as hacker interception tend to occur when using images as the only input. Hence, a safe and efficient offloading strategy is required.

  • (3)

    The state of art SLAM systems which can be used in robot embedded devices always have latency and cannot make sure the robustness of the system. After a long running, robots always lost themselves and cannot be restarted in a short time. In the condition of absence of any priori artificial calibration, how to achieve real-time calculation as well as ensure system robustness also need to be studied.

According to the above analysis, the main problems to solve are the insufficient on-board computing power and limited network transmission bandwidth, which cause multi-robot visual SLAM system very hard to achieve high robustness and real-time calculation. Therefore, we will tackle the aforementioned challenges with a two-step strategy. In the first step, we elaborate on the partial computing offloading method. Second, we propose a novel iterative closest point algorithm to reduce the computational complexity of multi-robot visual SLAM system.

The main contributions of our work can be summarized as follows.

  • (1)

    A novel partial offloading strategy: It is built to balance the calculation of local robots and cloud. To take full advantage of local computing and cloud computing, a strategy to offload most of the computation into the powerful cloud while reducing the amount of data which needs to be transmitted is designed. An efficient algorithm is proposed to find the best offloading point of the system. By this way, incomplete information is offloaded to the cloud in which case even if a hacker intercepts the network, it is hard for them to steal original and complete image information. In addition, this method greatly reduces the energy consumption as well as decreases the processing time.

  • (2)

    Improvement of point cloud map registration algorithm: The other main tasks of SLAM is map building, which is especially important for multi-robot SLAM system. The robots work independently in the environment and the maps built by each of them need to be matched. In order to reduce the complexity of the whole SLAM system, we improved classic registration algorithm, iterative closest point (ICP) algorithm, by proposing a novel fitness score hierarchical iterative closest point (FS-HICP) algorithm. The improved algorithm reduces the time costs and energy consumption.

  • (3)

    Construction of dense and semi-dense point cloud map: There are two kinds of map: the dense and semi-dense point cloud map, which are a set of data points in space and reflects the points on the external surfaces of objects [32]. The global dense map provides abundant visual and geometric information and is able to represent the environment for localization, path planning, virtual reality and so on [19]. On the other hand, a semi-dense point cloud constructs a sparse map containing limited information which describe partial features of the environment (general shape etc.) in application such as obstacle avoidance and path planning [26], [40]. In order to describe whether the scene is accessible, a semi-dense octree map is also constructed. And the two kinds of maps have different pros and cons so that they have different applications.

We also conduct extensive simulation and experimental evaluations to demonstrate the efficiency of the methods. Our partial offloading strategy and the FS-HICP algorithm can greatly reduce the whole system running time and the energy consumption.

The rest of this paper is organized as follows. Section 2 presents a brief survey of the related works. Section 3 presents the system model and then introduces the formulation of the partial offloading problem and multi-robot registration model. Section 4 discusses the details of the methods. Section 5 gives the evaluation results. Finally, Section 6 concludes the paper and presents our proposed future work.

Section snippets

Related works

Visual SLAM, as an effective and economical way to help robot localization and mapping in an unknown environment, has reached substantial robustness and accuracy in the centimeter range for single robot applications [7], [27]. ORB-SLAM2 [27] system proposed by Raul Mur-Artal et al. which uses three threads to work simultaneously, greatly improves the efficiency of the algorithm. And multi-robot systems have been becoming popular in numerous scenarios, ranging from search and rescue applications

System model and problem formulation

In this section, the total system framework is presented firstly. Then the model of multi-robot partial computing offloading is proposed. Finally, the multi-robot dense point cloud map model and its calculation method is introduced.

Method description

The objective of this section is to introduce our strategy to find the minimum value of Elk stated in Section 3, and then introduce an improved point cloud map registration algorithm, Fitness Score Hierarchical Iteration Closest Point(FS-HICP) algorithm, in order to reduce the value of Emapping.

Experimental results

This section presents the evaluation results of the improved methods through extensive simulation and experimental studies. The results include two parts: computing offloading and point cloud map fusion.

The embedded device used on the multi-robot system in this paper is Raspberry Pi 3 and data is transferred by Wi-Fi network. The RGB-D image acquisition device is Kinect 2.0 and the master node used in the simulation experiment is a desktop computer with 16 G memory and 4-core 3.3 GHz CPU. The

Conclusion and future work

Firstly, this paper proposes a multi-robot visual SLAM partial computing offloading strategy where the best offloading point is given to reduce the energy consumption and time cost of the whole visual SLAM system so that tasks that could not be achieved on the robot embedded device become possible. Secondly, an improved point cloud map registration algorithm (FS-HICP algorithm) suitable for indoor scene is proposed, which improves the convergence speed dramatically. Finally, we build the global

CRediT authorship contribution statement

Biwei Li: Conceptualization, Methodology, Software, Visualization, Data curation, Writing - original draft. Zhenqiang Mi: Supervision, Validation, Funding acquisition, Project administration. Yu Guo: Investigation, Resources. Yang Yang: Writing - review & editing. Mohammad S. Obaidat: Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

This work was supported in part by the National Natural Science Foundation of China under Grant 61772068, China Postdoctoral Science Foundation and Scientific and Technological Innovation Foundation of Shunde Graduate School, USTB .

Biwei Li received her B.S. degree in 2017 from University of Science and Technology Beijing, China. Now she is working towards the Master’s degree in the School of Computer and Communication Engineering at University of Science and Technology Beijing, China. Her main research interests include SLAM system, cloud computing and multi-robot systems.

References (40)

  • LeeS.M. et al.

    Dv-slam (dual-sensor-based vector-field slam) and observability analysis

    IEEE Trans. Ind. Electron.

    (2015)
  • SasaokaT. et al.

    Multi-robot slam via information fusion extended kalman filters

    IFAC PapersOnLine

    (2016)
  • ...
  • ArumugamR. et al.

    Davinci: A cloud computing framework for service robots

  • BenavidezP. et al.

    Cloud-based realtime robotic visual slam

  • BeslP.J. et al.

    Method for registration of 3-d shapes

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1992)
  • CadenaC. et al.

    Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age

    IEEE Trans. Robot.

    (2016)
  • FabresseF.R. et al.

    An efficient approach for undelayed range-only slam based on gaussian mixtures expectation

    Robot. Auton. Syst.

    (2018)
  • HanL. et al.

    Real-time global registration for globally consistent rgb-d slam

    IEEE Trans. Robot.

    (2019)
  • HuG. et al.

    Cloud robotics: architecture, challenges and applications

    IEEE Netw.

    (2012)
  • HunzikerD. et al.

    Rapyuta: The roboearth cloud engine

  • IndelmanV. et al.

    Multi-robot pose graph localization and data association from unknown initial relative poses via expectation maximization

  • IsobeY. et al.

    Occlusion handling for a target-tracking robot with a stereo camera

    Robomech J.

    (2018)
  • KochP. et al.

    Multi-robot localization and mapping based on signed distance functions

    J. Intell. Robot. Syst.

    (2015)
  • J. KUFFNER, Cloud-enabled humanoid robots, Humanoid Robots (Humanoids), 2010 10th IEEE-RAS International Conference on,...
  • KumarK. et al.

    Cloud computing for mobile users: Can offloading computation save energy?

    Computer

    (2010)
  • LabbeM. et al.

    Online global loop closure detection for large-scale multi-session graph-based slam

  • K. Lee, H. Kwon, K. You, Iterative solution of relative localization for cooperative multi-robot using iekf, 5(1)...
  • LeeH. et al.

    Probabilistic map merging for multi-robot rbpf-slam with unknown initial poses

    Robotica

    (2012)
  • LiQ.-s. et al.

    Building a dense surface map incrementally from semi-dense point cloud and rgbimages

    Front. Inf. Technol. Electron. Eng.

    (2015)
  • Cited by (6)

    • An automated multi-constraint joint registration method for mobile LiDAR point cloud in repeated areas

      2023, Measurement: Journal of the International Measurement Confederation
    • Cloud-based non-destructive characterization

      2023, Non-Destructive Material Characterization Methods
    • Research on Multi-Robot SLAM Map FusionMethod Based on Heuristics

      2022, Xitong Fangzhen Xuebao / Journal of System Simulation
    • AutoCIS: An Automated Construction Inspection System for Quality Inspection of Buildings

      2021, Proceedings of the International Symposium on Automation and Robotics in Construction

    Biwei Li received her B.S. degree in 2017 from University of Science and Technology Beijing, China. Now she is working towards the Master’s degree in the School of Computer and Communication Engineering at University of Science and Technology Beijing, China. Her main research interests include SLAM system, cloud computing and multi-robot systems.

    Zhenqiang Mi (S’09-M’11) received his B.S. degree in automation and Ph.D. degree in communication engineering, both from University of Science and Technology Beijing in 2006 and 2011, respectively. From 2015, he has been an associate professor with University of Science and Technology Beijing. His research interest includes service computing, multi-robot systems, and cloud computing in mobile environments.

    Yu Guo (S’ 16) received the B.S. and Ph.D. degrees from the University of Science and Technology Beijing, Beijing, China, in 2014 and 2020, respectively. He is currently a Postdoctoral Researcher with the School of Computer and Communication Engineering, University of Science and Technology Beijing. His main research interests include mobile wireless sensor and actor networks, cloud computing, and multi-robot systems.

    Yang Yang received his Ph.D. degree in information engineering from University of Science and Technology in Lillie, France, in 1988. He has been a professor of University of Science and Technology Beijing since 1988. His research interests include service science and cloud computing, image processing and pattern recognition, and grid technology.

    Professor Mohammad S. Obaidat (Fellow of IEEE 2005, Life Fellow of IEEE 2018) is an internationally known academic/researcher/scientist/ scholar. He received his Ph.D. degree in Computer Engineering with a minor in Computer Science from The Ohio State University, Columbus, USA. He has received extensive research funding and published To Date (2019) about One Thousand (1,000) refereed technical articles-About half of them are journal articles, over 95 books, and over 70 Book Chapters. He is Editor-in-Chief of 3 scholarly journals and an editor of many other international journals. He is the founding Editor-in Chief of Wiley Security and Privacy Journal. Moreover, he is founder or co-founder of 5 International Conferences.

    Among his previous positions are Advisor to the President of Philadelphia University for Research, Development and Information Technology, President and Chair of Board of Directors of the Society for Molding and Simulation International, SCS, Senior Vice President of SCS, Dean of the College of Engineering at Prince Sultan University, Chair and tenured Professor at the Department of Computer and Information Science and Director of the MS Graduate Program in Data Analytics at Fordham university, Chair and tenured Professor of the Department of Computer Science and Director of the Graduate Program at Monmouth University, Tenured Full Professor at King Abdullah II School of Information Technology, University of Jordan, The PR of China Ministry of Education Distinguished Overseas Professor at the University of Science and Technology Beijing, China and an Honorary Distinguished Professor at the Amity University- A Global University. He is now the Founding Dean of the College of Computing and Informatics at The University of Sharjah, UAE.

    He has chaired numerous (Over 160) international conferences and has given numerous (Over 160) keynote speeches worldwide. He has served as ABET/CSAB evaluator and on IEEE CS Fellow Evaluation Committee. He has served as IEEE CS Distinguished Speaker/Lecturer and an ACM Distinguished Lecturer. Since 2004 has has been serving as an SCS Distinguished Lecturer. He received many best paper awards for his papers including ones from IEEE ICC, IEEE Globecom, AICSA, CITS, SPECTS, DCNET International conferences. He also received Best Paper awards from IEEE Systems Journal in 2018 and in 2019 (2 Best Paper Awards). In 2020, he received 4 best paper awards from IEEE Systems Journal.

    He also received many other worldwide awards for his technical contributions including: The 2018 IEEE ComSoc-Technical Committee on Communications Software Technical Achievement Award for contribution to Cybersecurity, Wireless Networks Computer Networks and Modeling and Simulation, SCS prestigious McLeod Founder’s Award, Presidential Service Award, SCS Hall of Fame –Lifetime Achievement Award for his technical contribution to modeling and simulation and for his outstanding visionary leadership and dedication to increasing the effectiveness and broadening the applications of modeling and simulation worldwide. He also received the SCS Outstanding Service Award. He was awarded the IEEE CITS Hall of Fame Distinguished and Eminent Award.

    He is a Life Fellow of IEEE and a Fellow of SCS.

    View full text