Quantitative and Qualitative Evaluation of ROS-Enabled Local and Global Planners in 2D Static Environments

Abstract

Apart from perception, one of the most fundamental aspects of an autonomous mobile robot is the ability to adequately and safely traverse the environment it operates in. This ability is called Navigation and is performed in a two- or three-dimensional fashion, except for cases where the robot is neither a ground vehicle nor articulated (e.g. robotics arms). The planning part of navigation comprises a global planner, suitable for generating a path from an initial to a target pose, and a local planner tasked with traversing the aforementioned path while dealing with environmental, sensorial and motion uncertainties. However, the task of selecting the optimal global and/or local planner combination is quite hard since no research provides insight on which is best regarding the domain and planner limitations. In this context, current work performs a comparative analysis on qualitative and quantitative aspects of the most common ROS-enabled global and local planners for robots operating in two-dimensional static environments, on the basis of mission-centered and planner-related metrics, optimality and traversability aspects, as well as non-measurable aspects, such as documentation quality, parameterisability, ease of use, etc.

Appendices

Appendix A: Proportionality Contribution of Metrics to the Value of a Combination of Planners

On the value of a combination of global and local planners that pertains to metrics of its global planner component, we make the following assumptions. The value of a combination of global and local planners is: (a) higher the shorter a global plan is − so that a robot traversing it in constant speed takes less time to get from an initial to a goal pose, (b) higher the higher its resolution is − the finer a plan’s detail the more probable it is that a (sub)goal exists within the horizon of the local costmap, and the more smooth the path may be, (c) higher the more smooth it is − the smoother the plan the more probable it is that the robot takes less time to traverse the path from the initial pose to the goal one (following the global plan to the letter is a matter of the local planner as regards how fit it considers the global plan to be, and how feasible it actually is) (d) higher the larger its average mean minimum distance from obstacles in a map is − so that collisions with obstacles are less probable to occur (one must count on the uncertainty of improbable events) (e) higher the larger its overall minimum distance from obstacles in a map is across all simulations, and (f) lower the more varied the value of each metric is − so that the engineer can count on its reliability. As is evident, these metrics are independent of the success or failure of combinations of global and local planners in reaching the goal pose p_G from the initial one p₀, i.e. they are included in a combination’s calculation of value regardless if that combination failed to complete all missions.

On the value of a combination of global and local planners that pertains to metrics of its local planner component, we make the following assumptions. The value of a combination of global and local planners is: (a) lower the higher the mean number of aborted missions across the sum of its performed simulations is, (b) lower the higher the mean number of rotation recoveries performed is, (c) lower the higher the mean number of costmap clearances performed is, (d) lower the higher mean number of path failures exhibited is, (e) lower the higher the relative number of path failures is, and (f) lower the more varied the value of each metric is. Again, these metrics are independent of the success or failure of combinations of global and local planners in reaching the goal pose, and will therefore be included in a combination’s calculation of value regardless if that combination failed every single mission or not.

On the value of a combination of global and local planners that pertains to metrics referring to their joint performance, we make the following assumptions. The value of a combination of global and local planners is: (a) lower the larger the mean deviation of the actual paths the robot took − as a result of the application of the local planner − from the paths the global planner designed the robot to take, (b) lower the larger the mean total deviation of the former from the latter is, (c) lower the larger the mean Frechet distance of the former from the latter is, (d) higher the lower the travel time from the initial to the goal pose is, (e) higher the shorter the actual paths the robot took are, (f) lower the less smooth the actual paths the robot took are, (g) higher the larger the robot’s average mean minimum distance from obstacles in a map is, (e) higher the larger the robot’s overall minimum distance from obstacles in a map is across all simulations, and (f) lower the more varied the value of each metric is. The above metrics are dependent on the success or failure of the combination of global and local planners in reaching the goal pose, and will therefore not be included in the calculation of the value of a combination if that combination failed to reach p_G for every simulation that this occurred.

Appendix B: Details of Evaluation on Map CORRIDOR

1.1 1) Global-planner-specific evaluation

Tables 8 and 9 illustrate the resulted values of the quantitative metrics concerning global planners defined in Table 1 over N = 10 simulations in environment CORRIDOR.

Table 8 Mean global plan length $\mu _{l}(\boldsymbol {\mathcal {G}})$ and standard deviation $\sigma _{l}(\boldsymbol {\mathcal {G}})$, mean global plan resolution $\mu _{r}(\boldsymbol {\mathcal {G}})$, and mean $\mu _{s}(\boldsymbol {\mathcal {G}})$ and standard deviation $\sigma _{s}(\boldsymbol {\mathcal {G}})$ of the global plans’ smoothness for map CORRIDOR M_C

Table 9 Overall minimum distance of the global plans from any obstacle across all experiments $\inf (d(\boldsymbol {\mathcal {G}},\boldsymbol {M}_{C}))$, mean minimum distance $\mu (d(\boldsymbol {\mathcal {G}},\boldsymbol {M}_{C}))$ and standard deviation $\sigma (d(\boldsymbol {\mathcal {G}},\boldsymbol {M}_{C}))$ from all obstacles for map CORRIDOR M_C

Regarding the produced plans, global_planner produced paths of the least length (Table 8), sbpl_lattice_ planner those of the most length and of the lowest density but of the highest smoothness (lower numbers indicate higher smoothness), and navfn produced the least coarse plans but those of the lowest smoothness. The middle’s performance with respect to length is reasonable since sbpl_lattice_plannerdoes take account of the robot’s kinematic model, which, being a differential drive robot and therefore non-holonomic, is constrained in its motion. In contrast, navfn and global_planner do not consider such constraints and, the latter being the successor of the former, produce slightly similar plans (this is also observed when considering the figures of the two planners’ produced paths: they seem almost identical to the naked eye, in stark contrast to those of sbpl_lattice_planner). Another observable difference in Fig. ?? is that the plans of navfn and most of those of sbpl_lattice_planner are deterministic: given an initial pose p₀, a goal pose p_G and a map, they produce the same plan each time, while global_planner introduces a slight degree of randomness, which explains why its plans’ standard deviation is non-zero compared to the other two planners.

With regard to the crucial ability of a global planner to plan around obstacles (Table 9), global_planner produced paths that did not fully take account of the provided robot radius (subtracting the robot’s radius from the overall minimum distance of its plans to the closest obstacle gives − 0.02m), and therefore (a) a local planner of full fidelity to the global plan would, with certainty, force the robot to halt its mission (until perhaps a new target was set), or even to collide with obstacles in its environment and (b) the engineer should increase move_base’s view of the robot’s radius in its costmap parameters. The two remaining global planners produced paths that graze obstacles at least once. Furthermore, sbpl_lattice_planner would like to the robot to move parallel to walls, which is a behaviour that can actually be dictated to the planner (the planner was tuned so that the robot would prefer to move in straight lines), which can be considered somewhat advantageous since there is always an obstacle close enough so that it can be exploited as a frame of reference under the task of either mapping or localisation.

1.2 2) Local-planner-specific evaluation

Table 10 summarises the resulted values of the quantitative metrics concerning local planners defined in Table 1 over N simulations.

Table 10 Mean number of aborted missions over the number of simulations conducted μ_A/N, mean number of rotation recoveries μ_RR and their standard deviation σ_RR, mean number of costmap clearances μ_CC and their standard deviation σ_CC, mean number of path failures μ_PF and their standard deviation σ_PF, and mean number of path failures over the mean number of local planner calls μ_PF/μ_LPC for all combinations of global and local planners featured in Table 3 on map CORRIDOR M_C

All combinations of local planner dwa_local_ planner with global planners did not reach the goal pose even once, and this is due to the fact that it spent most of its time executing recovery behaviours (it has the highest mean rotation recoveries and costmap clearances among the three local planners), which resulted either in aborting the missions or in timing-out (ultimately, this is due to the fact that dwa_local_planner follows global plans with high fidelity, plans which are actually infeasible, since the overall minimum distance from obstacles is at most zero − Table 9). What is more is that it features the highest ratio of path failures (roughly one out of every ten times the local planner was invoked when the global planner used did not account for the robot’s kinematics, resulting in the former not being able to find valid motor inputs).

Local planner eband_local_planner performed better than dwa_local_planner: it never aborted a mission, it did not execute a single recovery behaviour, and it failed to procure valid motor commands with vastly lower frequency (although this planner does not offer access to its produced plans, and therefore there is no way of knowing how many times it was called to procure them, it is reasonable to assume that, given its low number of mean path failures, its failure frequency follows from the magnitude of the mean path failures, which was lower than that of dwa_local_planner). Its ultimate failure is that it is not fast enough (this can be observed in the mean travel times featured in Table 11; recount that $t_{C}^{max} = 120$ sec), i.e. its approach is overly safe. Its collaboration with global planner sbpl_lattice_planner was the worst, which could in theory be partially attributed to the fact that the latter produces the coarsest and lengthiest plans, but is in fact due to an unknown issue that causes the local planner to declare that the robot reached its goal while in fact still mid-way for some simulations (this is the second reason why sbpl_lattice_planner was given an inadequacy status in Table 2 − the first being the bug found and resolved that was mentioned in Section 5.1).

Table 11 Mean travel time μ_t from ${p_{0}^{C}}$ to ${p_{G}^{C}}$ and standard deviation σ_t for map CORRIDOR M_C

In contrast, local planner teb_local_planner championed in every available local-planner-specific metric: it never aborted a mission, it did not execute a single recovery behaviour, it never failed to procure valid motor inputs, and it never failed the robot in reaching the goal pose p_G.

1.3 3) Global/local-planners-combination-specific evaluation

Tables 11, 12, 13, and 14 summarise the resulted values of the quantitative metrics concerning the combination of global and local planners defined in Table 1 over N simulations for all combination of global and local planners in map M_C.

Table 12 Mean actual path length $\mu _{l}(\boldsymbol {\mathcal {P}})$ and standard deviation $\sigma _{l}(\boldsymbol {\mathcal {P}})$, and mean $\mu _{s}(\boldsymbol {\mathcal {P}})$ and standard deviation $\sigma _{s}(\boldsymbol {\mathcal {P}})$ of the actual path smoothness for map CORRIDOR M_C

Table 13 Overall minimum distance of the actual paths $\boldsymbol {\mathcal {P}}$ the robot took from any obstacle across all experiments $\inf (d(\boldsymbol {\mathcal {P}},\boldsymbol {M}_{C}))$, mean minimum distance $\mu (d(\boldsymbol {\mathcal {P}},\boldsymbol {M}_{C}))$ and mean standard deviation $\sigma (d(\boldsymbol {\mathcal {P}},\boldsymbol {M}_{C}))$ from all obstacles for map CORRIDOR M_C

Table 14 Mean deviation $\mu _{\delta }(\boldsymbol {\mathcal {P}},\boldsymbol {\mathcal {G}})$, mean total deviation $\mu _{\Delta }(\boldsymbol {\mathcal {P}},\boldsymbol {\mathcal {G}})$, and mean Frechet distance $\mu _{\delta }^{F}(\boldsymbol {\mathcal {P}},\boldsymbol {\mathcal {G}})$ between the actual paths $\boldsymbol {\mathcal {P}}$ the robot took and their corresponding global plans $\boldsymbol {\mathcal {G}}$ for map CORRIDOR M_C

In terms of time taken to reach the goal pose ${p_{G}^{C}}$ from ${p_{0}^{C}}$ (Table 11), all combinations of global planners with dwa_ local_planner are excluded from evaluation (since it’s a condition that the robot reach its goal), and the same applies to the combination of sbpl_lattice_planner and eband_local_planner. The remaining combinations illustrate (a) that under teb_local_planner the robot takes the least amount of time to traverse the path from p₀ to p_G (which is to be expected, since it approaches the problem of navigation in terms of optimising with respect to time), consequently (b) that eband_local_planner is the slowest among the two − with a significant margin since it takes more than twice as much time to complete a mission −, and (c) that the former’s travel times are the most consistent. The fact that sbpl_lattice_planner produces plans of greater length − approximately 17% lengthier than those of the other two global planners (Table 8) − made its combination with teb_local_planner have an impact on the robot’s travel time from start to finish, with a magnitude of about over two seconds, which translates to roughly a 10% increase in travel times compared to those of teb_local_planner with navfn and global_planner.

In terms of the traversed paths’ mean length (Table 28), the same combinations of global planners with eband_local_planner and teb_local_planner make the robot travel greater lengths compared to their global plans: both approaches deform the global plan in order to gain more clearance from obstacles, and this is the reason why dwa_local_planner fails in every single simulation. Furthermore, the paths that teb_local_planner dictated were the longest but the most consistent, and the most consistent of them all were observed when navfn was used as the global planner, which is to be expected since the plans produced by it are deterministic. In terms of path smoothness, sbpl_lattice_planner’s combination with teb_local_planner exhibited the highest and least-varied path smoothness.

In terms of clearance with regard to obstacles in map M_C (Table 13), the combination of local planner eband_local_planner with global planner sbpl_ lattice_planner didn’t make the robot collide with obstacles, while its clearance was lower than that of teb_local_planner. The latter’s parameterisation feature regarding minimum clearance from obstacles (set at 0.10m) clearly played a significant role in the robot’s distancing from obstacles: at 0.18m the planner gave the robot the largest minimum clearance from obstacles among all other local planners and across all experiments (this was a further reason why it scored as much with regard to the qualitative metric of parameterisability − Table 2). Furthermore, the same is observed with regard to the mean distance of each robot pose to the closest obstacle in the map of world CORRIDOR, while its variation is the least (compared to combinations that did complete the mission). As an aside, local planner dwa_local_planner failed to avoid obstacles at least once across N simulations on both occasions where the corresponding global planner was ignorant about the robot’s kinematics. On the other hand, when sbpl_lattice_planner was employed as the global planner, the robot did not collide with an obstacle even once. Furthermore, its mean distance to the closest obstacle was the highest among the three global planners employed, while its standard deviation was the lowest.

In terms of the traversed paths’ deviation from their corresponding global plans (Table 14), local planner dwa_local_planner exhibited the lowest mean pose deviation for its combinations with global planners that do not account for the robot’s kinematics, which is to be expected, since, as discussed earlier, it is the controller of highest fidelity to the global plan among the three. However, due to its inability to complete the mission, all of its combinations with global planners are excluded from further evaluation, and the same applies to its combinations with sbpl_lattice_planner and eband_local_planner. From the rest of the combinations, those employing teb_local_planner feature the least mean deviation of each pose with respect to the global plan, and, in particular, its combination with sbpl_lattice_planner exhibited the least total deviation among all other combinations. Furthermore, its mean discrete Frechet distance was consistently lower than that of its rival eband_local_planner.

Appendix C: Details of Evaluation on Map WILLOWGARAGE

1.1 1) Global-planner-specific evaluation

Tables 15 and 16 illustrate the resulted values of the quantitative metrics concerning global planners defined in Table 1 over N = 10 simulations in environment WILLOWGARAGE.

Table 15 Mean global plan length $\mu _{l}(\boldsymbol {\mathcal {G}})$ and standard deviation $\sigma _{l}(\boldsymbol {\mathcal {G}})$, mean global plan resolution $\mu _{r}(\boldsymbol {\mathcal {G}})$, and mean $\mu _{s}(\boldsymbol {\mathcal {G}})$ and standard deviation $\sigma _{s}(\boldsymbol {\mathcal {G}})$ of the global plans’ smoothness for map WILLOWGARAGE M_W

Table 16 Overall minimum distance of the global plans from any obstacle across all experiments $\inf (d(\boldsymbol {\mathcal {G}},\boldsymbol {M}_{W}))$, mean minimum distance $\mu (d(\boldsymbol {\mathcal {G}},\boldsymbol {M}_{W}))$ and standard deviation $\sigma (d(\boldsymbol {\mathcal {G}},\boldsymbol {M}_{W}))$ from all obstacles for map WILLOWGARAGE M_W

Regarding the produced plans (Table 15), and with respect to the metrics concerning the evaluation of global planners nothing changed compared to those concerning map M_C: global_planner again produced paths of the least length, sbpl_lattice_planner those of the most length and of the lowest density but of the highest smoothness, and navfn produced the least coarse plans, but those of the lowest smoothness.

What did change was the overall minimum distance from obstacles of the plans of global planner sbpl_lattice_planner (Table 16): while in the map of world CORRIDOR it never once planned through obstacles, in the map of world WILLOWGARAGE it did − as did global_planner again. Except for this, it again exhibited the lowest mean minimum distance from obstacles and the most consistency around it. On the other hand, navfn gained, on average, a centimeter of clearance, and its performance with respect to the mean minimum distance of each pose from obstacles was somewhat equivalent to that of global_planner’s, as was also its standard deviation.

1.2 2) Local-planner-specific evaluation

Table 17 summarises the resulted values of the quantitative metrics concerning local planners defined in Table 1 over N simulations.

Table 17 Mean number of aborted missions over the number of simulations conducted μ_A/N, mean number of rotation recoveries μ_RR and their standard deviation σ_RR, mean number of costmap clearances μ_CC and their standard deviation σ_CC, mean number of path failures μ_PF and their standard deviation σ_PF, and mean number of path failures over the mean number of local planner calls μ_PF/μ_LPC for all combinations of global and local planners featured in Table ?? on map WILLOWGARAGE M_W

The increased level of navigation arduousness of the WILLOWGARAGE world exposed more of the local planners’ shortcomings. What is impressive is that all combinations of global planners with local planners dwa_local_planner and eband_local_planner failed to traverse the path from the initial to the goal pose in all conducted simulations.

When global planners that do not account for the robot’s kinematics were used, the former aborted all missions, and it aborted most of them (7 out of 10) in the contrary case (− it becomes clear that the use of a global planner mindful of the robot’s kinematics is advantageous in the case of an inflexible local planner). In the former case it again exhibited the highest number of control failures and, in the latter, the highest mean number of costmap clearances.

With regard to the performance of eband_local_ planner in map M_W, the same as in map M_C applies for it in the case of its combination with sbpl_lattice_planner: it exhibited the lowest number of path failures (at least three times lower than the next lower-most combination). Although eband_local_planner managed to make the robot travel significantly larger distances compared to dwa_local_planner (Fig. 8), and although it was consistent in its navigation (the software bug mentioned in the previous subsection concerning its combination with sbpl_lattice_planner did not emerge in map M_W), it required more than the specified amount of time on every simulation, again exhibiting its overly-safe approach (its mean times were consistently slow, as previously observed in its simulations in map M_C). Its mean travel times are illustrated in Table 18 (recount that $t_{W}^{max} = 180$ sec).

Table 18 Mean travel time μ_t from ${p_{0}^{W}}$ to ${p_{G}^{W}}$ and standard deviation σ_t for map WILLOWGARAGE M_W

In contrast to all combinations of global planners with dwa_local_planner and eband_local_planner, all combinations with teb_local_planner managed to travel a path from the initial to the goal pose. Again, it championed in not aborting a mission even once, in reaching the goal pose on all simulations, in not executing a single recovery behaviour, and only its combination with global planner sbpl_lattice_planner failed to procure valid motor inputs, but only minimally (− the use of a global planner mindful of the robot’s kinematics seems not to be as advantageous in the case of a flexible local planner as it is in that of an inflexible one).

1.3 3) Global/local-planners-combination-specific evaluation

Tables 18, 19, 20, and 21 summarise the resulted values of the quantitative metrics concerning the combination of global and local planners defined in Table 1 over N simulations for all combination of global and local planners in the map of world WILLOWGARAGE.

/ Mean actual path length $\mu _{l}(\boldsymbol {\mathcal {P}})$ and standard deviation $\sigma _{l}(\boldsymbol {\mathcal {P}})$, and mean $\mu _{s}(\boldsymbol {\mathcal {P}})$ and standard deviation $\sigma _{s}(\boldsymbol {\mathcal {P}})$ of the actual path smoothness for map WILLOWGARAGE M_W

Table 20 Overall minimum distance of the actual paths $\boldsymbol {\mathcal {P}}$ the robot took from any obstacle across all experiments $\inf (d(\boldsymbol {\mathcal {P}},\boldsymbol {M}_{W}))$, mean minimum distance $\mu (d(\boldsymbol {\mathcal {P}},\boldsymbol {M}_{W}))$ and mean standard deviation $\sigma (d(\boldsymbol {\mathcal {P}},\boldsymbol {M}_{W}))$ from all obstacles for map WILLOWGARAGE M_W

Table 21 Mean deviation $\mu _{\delta }(\boldsymbol {\mathcal {P}},\boldsymbol {\mathcal {G}})$, mean total deviation $\mu _{\Delta }(\boldsymbol {\mathcal {P}},\boldsymbol {\mathcal {G}})$, and mean Frechet distance $\mu _{\delta }^{F}(\boldsymbol {\mathcal {P}},\boldsymbol {\mathcal {G}})$ between the actual paths $\boldsymbol {\mathcal {P}}$ the robot took and their corresponding global paths $\boldsymbol {\mathcal {G}}$ for map WILLOWGARAGE M_W

In terms of time taken to reach the goal pose ${p_{G}^{W}}$ from ${p_{0}^{W}}$ (Table 18), all combinations of global planners with dwa_local_planner and eband_local_planner are excluded from evaluation due to their inability to make the robot travel the whole path and reach ${p_{G}^{W}}$. − However, it should be noted that while dwa_local_planner’s combination with global planners ignorant of the robot’s kinematics could not even pass through the first challenging opening on walls (a door), its combination with sbpl_lattice_planner managed to do so, and do it for the next four openings, before caving in. The remaining combinations − all of them having teb_local_planner as their local planner − make the robot take just over half the allocated maximum time to travel the entirety of the path from ${p_{0}^{W}}$ to ${p_{G}^{W}}$. The fact that sbpl_lattice_planner produces plans of greater length − approximately 4.5% lengthier than those of the other two global planners (Table 15) − made its combination with teb_local_planner have an impact on the robot’s travel time from start to finish, with a magnitude of just over 5 seconds, which translates to roughly the same (5%) increase in travel times compared to those of teb_local_planner with navfn and global_planner.

In terms of the traversed paths’ mean length (Table 19), it is again observed that, due to the deformation of the global plan by teb_local_planner so that the set minimum obstacle clearance is achieved, the actual paths are longer than their corresponding global plans; and this to a degree of roughly 4.5% with regard to navfn and global_planner, and 1.5% with regard to sbpl_lattice_planner. In terms of path smoothness, its combination with navfn gave the smoothest paths, with the other two combinations following closely behind.

In terms of clearance with regard to obstacles in map M_W (Table 20), teb_local_planner just about managed to achieve the minimum set robot-obstacle distance (set as before to 0.10m) when combined with ROS’s default planners; when combined with sbpl_lattice_planner, however, it failed to do so. Its combination with global planner navfn gave the robot the largest mean clearance from obstacles across all simulations; its combination with sbpl_lattice_planner gave it the least mean clearance and the least variation around it (5cm less), a behaviour consistent with that exhibited in the map of world CORRIDOR.

In terms of the traversed paths’ deviation from their corresponding global plans (Table 21), no consistency is observed compared to the results regarding M_C, while teb_local_planner’s combination with global_planner exhibited the largest mean, total mean deviation and mean discrete Frechet distance in M_C, in M_W it exhibited the least; and its combination with sbpl_lattice_planner gave the largest mean discrete Frechet distance, when in M_C it gave the least.

1.4 D. Details of evaluation on map CSAL

1.4.1 1) Global-planner-specific evaluation

Tables 22 and 23 illustrate the resulted values of the quantitative metrics concerning global planners defined in Table 1 over N = 10 real-world experiments in environment CSAL.

Regarding the produced plans (Table 22), and with respect to metrics metrics concerning the evaluation of global planners, navfn produced paths of the least length, sbpl_lattice_planner those of the most length, of the lowest density, and of the highest variability, but of the highest smoothness; and global_planner produced the least coarse plans.

Table 22 Mean global plan length $\mu _{l}(\boldsymbol {\mathcal {G}})$ and standard deviation $\sigma _{l}(\boldsymbol {\mathcal {G}})$, mean global plan resolution $\mu _{r}(\boldsymbol {\mathcal {G}})$, and mean $\mu _{s}(\boldsymbol {\mathcal {G}})$ and standard deviation $\sigma _{s}(\boldsymbol {\mathcal {G}})$ of the global plans’ smoothness for map CSAL M_L

Global planner sbpl_lattice_planner exhibited the same behaviour as that in simulated world WILLOGARAGE: the overall minimum distance from obstacles of its plans (Table 23) was zero, and so was that of global_planner, which in all three study cases consistently planned through obstacles. Except for this, sbpl_lattice_planner exhibited again the lowest mean minimum distance from obstacles and the most consistency around it. On the other hand, navfn produced the best plans with respect to mean distance from obstacles, but those of the largest inconsistency among the three global planners.

Table 23 Overall minimum distance of the global plans from any obstacle across all experiments $\inf (d(\boldsymbol {\mathcal {G}},\boldsymbol {M}_{L}))$, mean minimum distance $\mu (d(\boldsymbol {\mathcal {G}},\boldsymbol {M}_{L}))$ and standard deviation $\sigma (d(\boldsymbol {\mathcal {G}},\boldsymbol {M}_{L}))$ from all obstacles for map CSAL M_L

1.4.2 2) Local-planner-specific evaluation

Table 24 summarises the resulted values of the quantitative metrics concerning local planners defined in Table 1 over N simulations.

What is impressive here is that all combinations of global planners with local planner dwa_local_planner again failed to traverse the path from the initial to the goal pose in all conducted experiments.

Table 24 Mean number of aborted missions over the number of simulations conducted μ_A/N, mean number of rotation recoveries μ_RR and their standard deviation σ_RR, mean number of costmap clearances μ_CC and their standard deviation σ_CC, mean number of path failures μ_PF and their standard deviation σ_PF, and mean number of path failures over the mean number of local planner calls μ_PF/μ_LPC for all combinations of global and local planners featured in Table 3 on map CSAL M_L

With regard to the performance of teb_local_ planner in environment CSAL, it again championed − never aborting any mission, and making no recovery attempts, although it experienced a minimal number of path failures. As for eband_local_planner, it exhibited minimal recovery attempts, but significant path failures when paired with global planners that do not account for the robot’s kinematics.

1.4.3 3) Global/local-planners-combination-specific evaluation

Tables 25, 26, 27, and 28 summarise the resulted values of the quantitative metrics concerning the combination of global and local planners defined in Table 1 over N simulations for all combination of global and local planners in environment CSAL.

Table 25 Mean travel time μ_t from ${p_{0}^{W}}$ to ${p_{G}^{L}}$ and standard deviation σ_t for map CSAL M_L

Table 26 Mean actual path length $\mu _{l}(\boldsymbol {\mathcal {P}})$ and standard deviation $\sigma _{l}(\boldsymbol {\mathcal {P}})$, and mean $\mu _{s}(\boldsymbol {\mathcal {P}})$ and standard deviation $\sigma _{s}(\boldsymbol {\mathcal {P}})$ of the actual path smoothness for map CSAL M_L

Table 27 Overall minimum distance of the actual paths $\boldsymbol {\mathcal {P}}$ the robot took from any obstacle across all experiments $\inf (d(\boldsymbol {\mathcal {P}},\boldsymbol {M}_{L}))$, mean minimum distance $\mu (d(\boldsymbol {\mathcal {P}},\boldsymbol {M}_{L}))$ and mean standard deviation $\sigma (d(\boldsymbol {\mathcal {P}},\boldsymbol {M}_{L}))$ from all obstacles for map CSAL M_L

Table 28 Mean deviation $\mu _{\delta }(\boldsymbol {\mathcal {P}},\boldsymbol {\mathcal {G}})$, mean total deviation $\mu _{\Delta }(\boldsymbol {\mathcal {P}},\boldsymbol {\mathcal {G}})$, and mean Frechet distance $\mu _{\delta }^{F}(\boldsymbol {\mathcal {P}},\boldsymbol {\mathcal {G}})$ between the actual paths P the robot took and their corresponding global plans G for map CSAL M_L

In terms of time taken to reach the goal pose ${p_{G}^{L}}$ from ${p_{0}^{L}}$ (Table 25), all combinations of global planners with dwa_local_planner are excluded from evaluation due to their inability to make the robot travel the whole path and reach ${p_{G}^{L}}$. Local planner teb_local_planner traversed the assigned paths in the less amount of mean time compared to eband_local_planner for the same global planner.

In terms of the traversed paths’ mean length (Table 26), teb_local_planner did not deform its given global plans to the degree it did in simulations, and this is observable as it produced paths with the least mean length. On the other hand, eband_local_planner’s dictated paths were the longest, but most consistent in length. The latter produced paths of lower smoothness, compared to the former, and those of the most consistency with regard to smoothness.

In terms of clearance with regard to obstacles in map M_L (Table 27), teb_local_planner did not manage to achieve the minimum set robot-obstacle distance (set as before to 0.10m) when combined with any global planner (its mean value was 0.08m), however, under its control, the robot’s mean minimum distance to obstacles and its standard deviation around it were lower than those when local planner eband_local_planner was employed. The latter was the only local planner that did achieve to exhibit more than the set robot-obstacle threshold distance, and it did so consistently.

In terms of the traversed paths’ deviation from their corresponding global plans (Table 21), teb_local_ planner produced paths of the lowest mean and total deviation, which is consistent with the mean minimum robot-obstacle distance values exhibited. Conversely, eband_local_planner, consistent with its own mean minimum robot-obstacle distance, produced paths of the greatest mean and total deviation from global plans, and, overall, those of the largest Frechet distance.