An optimization-based selection approach of landing sites for swarm unmanned aerial vehicles in unknown environments

Highlights

• Landing sites selection for swarm UAVs using optimization models.

• A coarse-to-fine-based manner is adopted to reduce the computation complexity.

• Optimization landing model considering multiple constraints.

• Three scenes demonstrate the effectiveness in normal and emergency situations.

Abstract

Autonomous selection of landing sites is an important capability of a mission when unmanned aerial vehicles (UAVs) face technical difficulties or adverse weather, especially in unknown environments. In order to build a better decision-making system for swarm UAVs landing, besides terrain safety, the global flight cost among individuals should be also considered during descent. However, the existing methods cannot solve these problems elegantly due to terrain uncertainty and optimization complexity. In this paper, we present an optimization method to tackle this issue based on point cloud integrating the coarse-to-fine manner and swarm intelligence model. Specifically, our method starts with a point cloud preprocessing module for sparse elevation estimation and robust path planning. In the coarse stage, a novel cost map is constructed by extracting only low-level features from the elevation map. Based on the optimum results of the coarse selection, the final landing sites are automatically generated and finely evaluated in terms of multiple 3D terrain factors. Finally, the landing model of swarm-UAVs is treated as an unbalanced assignment problem, which minimizes flight costs by incorporating terrain safety, fuel consumption, and path planning. Experimental evaluations on three different real-world scenario datasets demonstrate the effectiveness of our method in both normal operations and emergency situations.

Introduction

Unmanned Aerial Vehicles (UAVs) have been significantly growing in recent years, due to their extraordinary versatility and increasing robustness (Shah Alam and Oluoch, 2021, Chen et al., 2020). In comparison to the single UAV, swarm UAVs demonstrate more superior performance in coordination, intelligence, and autonomy, since they are capable of accomplishing more complex tasks through efficient collaborative division and global decision-making among individuals (Chen et al., 2021, Schmitt and Stütz, 2017). Based on this, swarm UAVs are applied widely in both military and civilian areas, such as urban search and rescue (USAR) missions, target surveillance, communications networks, and geological exploration (Pi et al., 2020). Once the UAVs face extreme weather conditions, technical glitches, and life-saving after completing the mission, they need to land safely nearby no matter whether the landing scenes are known or unknown. However, landing site detection and the automatic safe landing still is an important and challenging issue, especially in unknown environments. This is because no prior information about the landing scenes, such as specific patterns, Digital Surface Maps (DSM), digital terrain models (DTM), and pre-existing maps, can be utilized for UAVs flight navigation (Mittal et al., 2019, Scherer et al., 2012). Furthermore, our paper aims to achieve autonomous landing of swarm UAVs at minimum flight cost in complex terrain, which makes this problem more difficult. In reality, safe landing at unprepared sites is essential for future UAV systems, which can effectively avoid accidents in both normal and emergency situations.

The autonomous landing of UAVs is the ability to detect several landing sites, which are relatively open, flat enough, sufficiently large, and free of obstacles. However, the previous UAVs landing systems excessively depended on wireless communication, prior map information, and pre-defined landing mark in the known scenarios, such as Global Positioning System (GPS) (Ayhan et al., 2018), radars (Shin et al., 2017), IR-LED (Infrared Light Emitting Diode) pattern (Wenzel et al., 2011), and H-shaped marker (Patruno et al., 2019, Chen et al., 2021, Yu et al., 2017). In this way, these works can easily obtain the relative position relationship between the UAV and landing site, because characteristics of the final landing locations are known in advance. However, these methods are not suited for autonomous UAVs landing in unknown or adverse environments, due to the no prior guidance from the navigation system and map information. In order to achieve autonomous landing of UAVs in unstructured scenes, many researchers are investigating actively this issue depending on vision-based and non-vision-based techniques.

In vision-based methods, the landing systems are defined differently due to the number of cameras used in the UAV. In terms of monocular vision, a representative framework is proposed to detect landing zone using semantic segmentation only from scene images. Under these onboard systems, researchers can use color, geometry, texture, or high-level feature to estimate candidate regions for UAV landing (Kaljahi et al., 2019, Yan et al., 2013, Rao et al., 2016). However, the common drawback of these methods is that they cannot obtain important terrain 3D surface conditions, such as slope and flatness. In particular, terrain analysis of landing sites is an important and valuable ability for UAV systems, which can greatly improve the automation level and avoid unnecessary risks. To this end, existing more vision-based methods have been developed over time for 3D terrain representation, such as Stereo ranging (Garg et al., 2018), Optical flow (Cheng et al., 2019, Cheng et al., 2019), Simultaneous Localization and Mapping (SLAM) (Hinzmann et al., 2018), Homography Estimation Adaptive Control (HEAC) (Bosch et al., 2006), and Structure from Motion (SFM) (Bi et al., 2017). All these methods generate a sparse or dense point cloud of the environment for UAVs landing by matching or tracking corresponding feature points from the 2D image sequences of moving cameras. Based on 3D data, the UAV can easily guide towards safe landing sites in unknown environments without any landmarks. Though these reconstruction methods can obtain 3D terrain conditions for determining safe landing sites of UAVs, they still need help from sub-systems, such as GPS and Inertial Navigation Sensors (INS). Furthermore, there exist inevitable errors in the image-based 3D reconstruction performances, especially in the complicated structure, low textured, and lighting areas (Gao et al., 2020). In non-vision-based techniques, the Light Detection and Ranging (LiDAR) is adopted widely to generate the dense point cloud of landing scene directly by registering multiple reference patches together (Scherer et al., 2012, Johnson et al., 2002, Lorenzo et al., 2017). After terrain analysis, several studies have been performed on LiDAR and find a safe landing zone from the target surface. Although LiDAR is a dependable sensor for 3D terrain evaluation, it is heavier and more costly compared to the widespread camera (Singh et al., 2015); Xiao et al., 2019). Therefore, the latest methods are paying more attention to vision-based techniques to find landing sites under light-load conditions. However, the impacts of LiDAR for vehicle landing should not be ignored in some important scenes, such as landing on Mars, planetary exploration(Johnson et al., 2002, Wu et al., 2021). Despite the data formats acquired by the passive or active sensor are different, the corresponding landing site strategy is similar, this is because all point clouds data formats are suitable for our method.

To find a safe landing site, several terrain factors which can be referred to (Schmitt & Stütz, 2017) are involved as follows: 1) UAVs size, slope, and roughness; 2) Terrain surface condition; 3) Obstacle height and distance. To further improve and optimize landing performance, besides terrain factors, other flight constraints should be also taken into consideration jointly in actual missions, such as touchdown performance, descent dynamics, surface type, fuel consumption, local wind field, and surrounding obstacles (Mittal et al., 2019, Hinzmann et al., 2018, Cui et al., 2017). In this way, the landing sites can be ensured according to certain priority levels by incorporating the above multiple factors. However, the current methods are not capable of solving this problem completely, which mainly manifested in three aspects: 1) The complexity of computation caused by huge data, especially dense point clouds. 2) The feasible evaluation criteria of the landing site. 3) The architecture optimization of landing site selection under multiple factors. In this paper, we present a novel approach to address autonomous selection of landing sites of swarm UAVs based on coarse-to-fine strategy and optimization algorithms. By building on the point cloud, our pipeline takes both terrain safety and fuel consumption into consideration. In swarm UAVs, besides the problems of landing site detection and path planning, global flight cost is ensured to be as low as possible during descent. To solve this problem, the landing sites selection of swarm UAVs is converted into an optimization problem. Furthermore, in contrast to recent methods which perform coarse-to-fine manner on the 3D space region with different spatial resolutions (Scherer et al., 2012, Zheng et al., 2021), the proposed method uses two stages in 2D cost map and 3D point cloud space respectively. In this way, only the most promising candidate landing sites are evaluated in the next stage, which achieves real-time operation. In addition, there exist very few papers that focus specifically on the autonomous selection of landing sites for swarm UAVs in unknown environments.

In latest years, deep learning methods have been used in several landing missions, due to their high semantics and scalability. In general, these papers are divided into two categories: one is depth estimation methods based on images acquired by moving camera (L. Chen et al., 2020), the other is confidence estimation methods based on occupancy map from 3D point cloud (Maturana and Scherer, 2015, Iiyama et al., 2021, Tomita et al., 2021). However, the deep learning-based method in actual landing missions is difficult and rarely applied. The major reason is that though there exist several labeled datasets, it still lacks reliable and large-scale real data for training. Furthermore, these methods do not have good generalization ability, resulting in limited performance in unknown environments.

In this paper, our approach adopts a coarse-to-fine strategy for final landing sites. In the coarse stage, we only use low-level image features from the 2D elevation grid-map to locate candidate landing regions quickly by setting the cost map. This cost map consists of obstacle distance, ground slope, surface conditions. Meanwhile, these landing structure features are approximated and quantified by designed distance transform, local plane difference, and gradient feature respectively. Next, followed by the sparse and optimum constraint, the candidate landing sites are generated dynamically. In this way, only the most promising landing sites are evaluated in finer-grained, which cut down the computation largely. We also investigate path planning of UAVs in the light-weight OctoMap for fuel consumption metric. In order to perform an optimum landing for UAVs swarm, the global landing cost using multiple optimization algorithms is also constructed and compared. This paper defines the problem of landing sites of swarm UAVs, describes the coarse-to-fine manner in different processing spaces and constructs the landing model of swarm UAVs at minimum flight cost.

The main contributions of our paper are threefold as follows:

We build a complete framework of swarm UAVs landing in unknown environments, that unifies four key modules: point cloud preprocessing, coarse cost-map generating, fine evaluation of landing site, and optimum landing model.

A coarse-to-fine-based evaluation manner is performed on a cost map and point cloud space respectively to further reduce the complexity of computation. Meanwhile, the low-level 2D image features and 3D terrain structure features are quantified and evaluated effectively by designed normalization and weighted cost functions.

This paper proposes an optimum landing model of swarm UAVs, which selects the final landing sites by assessing terrain safety, fuel consumption, and path planning using optimization methods.

Extensive experiments demonstrate that our method can obtain favorable landing performance in different real-world scenarios, including indoor scenes, outdoor vegetated scenes, and urban scenes.