Cooperative and Geometric Learning Algorithm (CGLA) for path planning of UAVs with limited information

Abstract

In this paper, we propose a new learning algorithm, named as the Cooperative and Geometric Learning Algorithm (CGLA), to solve problems of maneuverability, collision avoidance and information sharing in path planning for Unmanned Aerial Vehicles (UAVs). The contributions of CGLA are three folds: (1) CGLA is designed for path planning based on cooperation of multiple UAVs. Technically, CGLA exploits a new defined individual cost matrix, which leads to an efficient path planning algorithm for multiple UAVs. (2) The convergence of the proposed algorithm for calculating the cost matrix is proven theoretically, and the optimal path in terms of path length and risk measure from a starting point to a target point can be calculated in polynomial time. (3) In CGLA, the proposed individual weight matrix can be efficiently calculated and adaptively updated based on the geometric distance and risk information shared among UAVs. Finally, risk evaluation is introduced first time in this paper for UAV navigation and extensive computer simulation results validate the effectiveness and feasibility of CGLA for safe navigation of multiple UAVs.

Introduction

The Unmanned Aerial Vehicles (UAVs) have been widely applied in military fields and civil industries, and there are significantly growing research interests in this area over the past decade (Bortoff, 2008, Chandler and Pachter, 2002, Chandler and Rasmussen, 2000, Rabbath et al., 2004, Zheng et al., 2005). The maneuverability, collision avoidance and information sharing in path planning for Unmanned Aerial Vehicles (UAVs) are among the most challengeable problems. The current progress has focused on these challenges from single UAV to multiple UAVs (Bellingham et al., 2002, Flint and Fernandez, 2005, Kim et al., 2006, Nikolos et al., 2007, Wu et al., 2011).

In previous works, Voronoi Graph Search (Bortoff, 2008) and Visibility Graph Search (Bellingham et al., 2002) are among the earliest path planning methods for both single UAV and multiple UAVs, but they have been proven to be effective only in simple environment. Both methods suffer from a fatal failure when the map information is partially available, especially when some obstacles are not detected. The A2D method reported in Kim et al. (2006) can be considered as a rule and inference based approach. When the environment becomes complicated, the method failed to find a good path to escape from dangerous zone. Evolutionary algorithms (Nikolos et al., 2007) have been used as a viable candidate to effectively solve path planning problems and can provide feasible solutions within a short time. Being different with a single UAV, the path planning for multiple UAVs mainly concentrates on the collaborative framework, information sharing (Bauso et al., 2004, Beard and Stepanyan, 2003, Dogman, 2003, Flint and Fernandez, 2005, Flint et al., 2002, Girard et al., 2007, Shanmugavel et al., 2010). Researchers in Bellingham et al. (2002) use the Mixed-integer Linear Programming to find a global path for multiple UAVs in order to avoid collision. However, the method fails to handle sudden changes in a local region. It is widely known that Dynamic Programming (DP) can be used to obtain an optimal path when full information and unlimited computation resource are available (Flint & Fernandez, 2005). In Flint et al. (2002), a dynamic programming method is presented to produce the near-optimal trajectories for multiple UAVs to cooperatively search for targets. However, path planning for multiple UAVs is generally a problem depending on individual behaviors, which is not directly suitable for global methods such as dynamic programming. A number of the improved versions of DP have since then been proposed. For example, a so-called stochastic dynamic programming is proposed for UAV path planning in Girard et al. (2007). The method is actually based on random parameters, specifically the number of flying times. The cost function is one of the key issues in DP and its variants. The distance between the center mass and the target is used in Bauso et al. (2004), and the distance and cooperation measures are exploited in Flint and Fernandez (2005) for UAV path planning. In Dogman (2003), the authors propose a path planning algorithm based on a map with probability of threats, which is built from a priori surveillance data. In Bauso et al. (2004) and Beard and Stepanyan (2003), the researchers develop a novel hybrid model, and design consensus protocols for the management of information. They further synthesize local predictive controllers through a distributed, scalable and suboptimal neuro-dynamic programming algorithm. Fuzzy reasoning, or approximate reasoning, has been an active topic in the fuzzy community since the inception of Zadeh’s pioneering work (Zadeh, 1974). In Zhao, Zheng, Liu, Cai, and Lin (2011) and Zheng, Wu, Liu, and Cai (2011), fuzzy reasoning methods treat fuzzy reasoning as a process of optimization rather than logical inference in the UAV path planning. An explicit feedback mechanism and the ‘virtual obstacle’ method, the so-called feedback based CRI (FBCRI), are embedded into the optimal fuzzy reasoning method to solve the path planning for multiple UAVs (Zhao et al., 2011, Zheng et al., 2011). In FBCRI, the fuzzy rule base is updated during the reasoning process by incorporating newly generated rules into the original rule base (Zheng et al., 2011). Furthermore, by embedding some virtual sub-goals into the FBCRI based approach, a new cooperative path planning approach based on virtual sub-goals (CPVS) is proposed in Zhao et al. (2011). But the analytic robustness analysis of various fuzzy reasoning methods is often very complicated (Cai and Zhang, 2008, Zheng et al., 2011). Though there has been significant progress on path planning for multiple UAVs based on a given risk map and logical feedback controller, to the best of our knowledge, the path planning for multiple UAVs based on adaptive information sharing is not well studied in the literature.

In this paper, we address the path planning problem for both single and multiple UAVs in a dynamically changing environment from perspectives of information sharing and reinforcement learning (Even-Dar and Mansour, 2003, Watkins, 1989, Zhang, Mao, Liu, Liu, and Zheng, 2013). In path planning, $Q$-Learning (Watkins, 1989) (reinforcement learning) is of high performance for the case when the entire map is known to planners (Mao et al., 2012, Zhang, Mao, Liu, Liu, and Zheng, 2013). Geometric distance and information sharing are believed to be very important elements for path planning when partial area of the map is changing (Zhang, Mao, Liu, & Liu, 2013) based on a shared weight matrix and the reward matrix by all UAVs in a random process. However, such a valuable information is not used by the traditional reinforcement approach. In the theoretical aspect, we know that the convergence of $Q$-Learning is only studied in a probabilistic manner (Even-Dar & Mansour, 2003). Motivated by the above observation, we propose a new algorithm, called the Cooperative and Geometric Learning Algorithm (CGLA), to build a general path-planning model based on the criterion in terms of geometric distance and integral risk information. The theoretical convergence and complexity analysis about the proposed method are also well investigated in this paper.

The concepts of the Individual Weight Matrix and cost matrix are proposed in this paper and they lead to an efficient path planning of both single UAV and multiple UAVs. Both matrices can be dynamically updated by taking both the distance information and integral risk into consideration. A single weight matrix or cost matrix is used by all UAVs (Zhang, Mao, Liu, & Liu, 2013), which will result into bad path planning results when no communication is available among UAVs, even can be of higher efficiency. This paper exploits an individual weight matrix and the cost matrix for a more reasonable path planning. In multiple UAVs, the “virtual obstacle” is proposed and embedded into the weight matrix calculation for prevention of possible collisions. Extensive computer simulation results show that the proposed approach performs very well in terms of the path length and the integral risk measure, which was validated to be a good criterion for path planning.

The rest of the paper is organized as follows: Section  2 describes the UAV threat environment modeling. Sections  3 The Individual Weight Matrix, 4 Cooperative and geometric learning algorithm present the main components of the CGLA algorithm. The extensive computer simulation results are given in Section  5, and Section  6 concludes the paper.