skip to main content
research-article
Open access

Point Set Registration for Target Localization Using Unmanned Aerial Vehicles

Published: 13 May 2023 Publication History

Abstract

The problem of point set registration (PSR) on images obtained using a group of unmanned aerial vehicles (UAVs) is addressed in this article. UAVs are given a flight plan each, which they execute autonomously. A flight plan consists of a series of GPS coordinates and altitudes that indicate where the UAV stops and hovers momentarily to capture an image of stationary targets on ground. A PSR algorithm is proposed that, given any two images and corresponding GPS coordinates and altitude, estimates the overlap between the images, identifies targets in the overlapping area, and matches these targets according to the geometric patterns they form. The algorithm estimates the overlap considering the error in UAVs’ locations due to wind, and it differentiates similar geometrical patterns by their GPS location. The algorithm is evaluated using the percentage of targets in the overlapping area that are matched correctly and the percentage of overlapping images matched correctly. The target-matching rate achieved using only the GPS locations of targets varied from 44% to 55% for target densities that varied from 6.4 down to 3.2 targets/m2. The proposed algorithm achieved target-matching rates of 48% to 87%. Well-known algorithms for PSR achieved lower rates on average.

1 Introduction

The use of unmanned aerial vehicles (UAVs) for monitoring areas of difficult and/or costly access to humans and/or manned aircraft has expanded in recent years. UAVs are now used for varied services including public safety, surveillance, inventory, inspection of infrastructure, and agriculture. In many of these application scenarios, UAVs capture several images that partially overlap the area to be inspected. Several images are needed because of a proximity requirement from the UAVs to the surface in most cases. A single image that covers all of the area of interest is not sufficient because UAVs would have to be distant from the surface, and the information to be extracted from the image would not be captured due to insufficient resolution. As images are captured, they need to be assembled into a mosaic so that overlapping areas between adjacent images are aligned and matched correctly. This alignment problem is known in the literature as point set registration (PSR). In PSR, two point sets are given, one per image, and the objective is to find a spatial transformation that aligns the two point sets.
Therefore, this article considers the PSR problem for images obtained from UAVs that cover an area of interest with images that overlap, partially forming a mosaic. The partial overlap is needed because the UAVs’ location, tilt, and height are randomly affected by wind. Without the partial overlap of images, the probability of sections of the area of interest not being covered by any image increases to unacceptable levels. The alignment of images is randomly affected by wind, so the objective is to realign the images so that targets that are covered by more than one image are identified correctly. Otherwise, a single target on the ground may be interpreted as several different targets on the mosaic.
This article is a continuation of our preliminary work in [13], and it is motivated by recent work on identification of static targets in aerial images such as parked cars [20, 28, 40], palm trees [7], plants with foliar disease [41], birds’ nests [21], and targets from a static camera such as body appearance [34], head-shoulders [48], persons [10], cars [31], and crowd size [17, 38, 49]. However, the algorithms developed in [7, 10, 17, 20, 21, 28, 31, 34, 38, 40, 41, 48, 49] are for target detection in one image or an array of images already aligned. On the other hand, current and well-known PSR algorithms, which will be reviewed in Section 2, are not designed for arrays of images obtained from UAVs. Images from UAVs include information on geographical location and direction not considered by current and well-known PSR algorithms. Also, current PSR algorithms are not aimed at small sets of targets; they are aimed at large sets of points that form clear shapes such as human faces to be identified. In our case, the number of targets in one image is small relative to the well-known PSR algorithms, and our targets are randomly placed on the ground so they do not form a clearly defined global pattern that can be identified. As demonstrated in Section 5, current and well-known PSR algorithms do not perform well for the type of target patterns considered in this article.
The organization of this article is the following. The related work and contributions are presented in Section 2. A description of our system is provided in Section 3 including the fly zone, UAVs, flight path, target distributions, image dimensions, and overlap between images. The proposed PSR algorithm is described in Section 4. The experimental procedure to evaluate the performance of our algorithm and a comparison of this performance against four well-known and influential PSR algorithms are presented in Section 5. A discussion of the experimental results is done in Section 6. The article is concluded in Section 7.

2 Related Work

Traditionally, the problem addressed in UAVs is the registration of images to form a mosaic of a larger panoramic image in which there are no specific targets to be detected (e.g., cars, persons, plants) but features such as in cartography. There are four steps in this problem: feature extraction, feature matching, transformation estimation, and blending. The goal of feature extraction is to detect features (e.g., line segments, corners, planes) that remain constant across one or more image transformations such as translation, rotation, or scaling. Representative feature detectors include scale-invariant feature transform (SIFT) [24], speeded-up robust features (SURF) [6], features from accelerated segment test (FAST) [36], and oriented FAST and rotated BRIEF (ORB) [37]. Feature matching and transformation estimation are usually performed via random sample consensus (RANSAC) [15] to remove outliers from the features extracted or via PSR algorithms such as iterative closest point (ICP) [8], coherent point drift (CPD) [29], Gaussian mixture models (GMMs) [22], locality-preserving matching (LPM) [26], or linear assignment [23].
Examples of registration of images obtained from UAVs include [9, 11, 18, 30, 32, 33, 45, 46, 47]. In [9], the images are used for determining the location and orientation of the UAV. This problem is solved by doing the registration of the images with respect to multiple LiDAR scans of the urban environment. Features are detected and registered using RANSAC [15] and ICP [8]. In [33], pairwise image registration is done with GeoMI, a proprietary software that uses cross-correlation. The problem addressed is the rapid blending of very large numbers of paired images. In [32], 3D images obtained from lasers on a UAV are registered using RANSAC [15] and ICP [8] with the objective of detecting surfaces and boundaries to avoid collisions. In [47], the objective is to perform registration of aerial images of arable land. Features are extracted using SURF [6], and pairwise image registration is done via mixture models similar to GMMs in [22]. In [11], the objective is to perform registration of images of an urban panoramic view. Features are extracted using ORB [37]; feature matching is done via LPM [26], and the transformation is estimated using CPD [29]. In [18], the objective is to perform registration of aerial images of urban environments. Features are extracted using SURF [6], and pairwise image registration is done via RANSAC [15]. In [30], the objective is to perform registration of aerial images of rural and urban environments. Features are extracted using SIFT [24], and feature matching is done via linear assignment [23]. In [46], the objective is to compare the performance of four different transformations after feature detection is done via SIFT [24] and outliers are removed using RANSAC [15]. The transformations considered include affine, quadratic polynomial, Delaunay triangulation, and homography matrix, with the affine transformation performing best. In [45], the objective is to perform registration of satellite images of rural areas. Feature extraction and matching are performed using SIFT [24] and RANSAC [15].
In the previous image registration scenarios, feature detection is needed because there are not clearly defined targets to be detected such as cars, plants, or people [7, 10, 17, 20, 21, 28, 31, 34, 38, 40, 41, 48, 49], which is the case considered in this article. Therefore, this article implements target detection instead of feature extraction followed by pairwise image registration, and the focus is on the PSR problem using the detected targets. The literature on PSR has generally focused on matching large sets of points on two images that represent complex patterns such as human faces, which is not the case considered in this article. Targets in our article are randomly located without following any particular general pattern as in [7, 10, 17, 20, 21, 28, 31, 34, 38, 40, 41, 48, 49]. Current and influential PSR algorithms include [8, 22, 26, 29], which have been applied to images obtained with UAVs as reviewed previously, and [5, 14, 19, 25, 39, 43, 44, 50], which have not been applied to UAV images to the best of our knowledge. These algorithms are reviewed next on the basis of two influential algorithms, ICP [8] and CPD [29].
The ICP algorithm [8] registers 3D shapes including free-form curves and surfaces. It depends on the monotonic convergence to the nearest local minimum of a mean-square distance metric, so ICP is vulnerable to initial conditions (i.e., initial alignment between the two images). It assumes that in the initial alignment, every closest point pair corresponds to each other. Improvements to ICP took place in [5, 14, 19, 43, 44]. In [14], ICP is modified to register images of different scale [14]. In [19], ICP is modified to make the registration of point clouds less susceptible to initial conditions and outlier points (e.g., noise). This is achieved by minimizing the Kullback-Leibler divergence of the probability distribution of one of the point clouds and the probability distribution of the transformation of the other point cloud. The same improvement is addressed in [44] via minimization of the correntropy between the point set of one of the images and the point set of a rigid transformation of the second image. The problem of multiple-view registration using ICP is addressed in [43]. The objective is to perform a global (i.e., multiple view) registration of images of the same object obtained from different viewpoints, so this is not a pairwise registration but a simultaneous registration of all viewpoint images. The problem of aligning a dense point cloud with a sparse one is addressed in [5] via a technique called probabilistic data association (PDA). This technique associates each point in the source point cloud with a set of points in the target point cloud, and each association is weighted so that weights form a probability distribution. Further development of probabilistic association was accomplished with the FilterReg algorithm proposed in [16].
The CPD algorithm [29] registers point sets within an arbitrary number of dimensions, and it works by fitting the GMM centroids of one point set to the other via expectation maximization. In [50], CPD is applied to images obtained via synthetic aperture radar. The images are first processed via SURF [6] to extract features, which are edges in this case, and CPD is then applied to the SURF images to register the SURF edges. An accelerated version of CPD is developed in [25] due to its high complexity. The PSR algorithm in [22] uses GMMs and operates on principles similar to the use of GMMs in CPD. The point set of each image is represented as a GMM, and the \(\text{L}_\text{2}\) distance between the GMM of one of the point sets and the GMM of the transformation of the other point set is minimized. Another similar approach is realized in [39] using a subclass of GMMs, which is 3D Normal Distribution Transform (NDT) models.
Deep-learning has also been applied to the problem of point set registration. For example, the neighborhood-consensus method proposed in [35] uses the PF-Pascal dataset, which contains 1,351 semantically related image pairs from 20 object categories. The method uses 700, 300, and 300 image pairs for training, validation, and testing, respectively. Another deep-learning method was proposed in [42] and uses a smaller dataset of point clouds for training. We refer to this method as ICDO. Training takes place with seven point-cloud sets that are artificially modified with random noise and rotations to generate a larger set of point clouds for training.
The contributions of this article are as follows:
Two UAVs were programmed to fly autonomously and capture images of targets on the ground within an area of interest. The programmed flight path considers the deviations on UAVs’ trajectories due to wind in order to avoid sections of the area not being covered by any image.
A PSR algorithm that considers the geographical location of images from UAVs is developed for images of discrete targets randomly distributed on the ground. Targets are not assumed to follow any particular global pattern.
The proposed PSR algorithm is evaluated using images obtained with two UAVs. The performance is reported in terms of the percentage of targets that are matched correctly within the overlapping sections of adjacent images. This percentage decreased from 87% to 48% when the density of targets increased from 3.2 to 6.4 targets/m2. The performance is also evaluated in terms of the percentage of images matched correctly, which decreased from 83% to 31%.
The performance of the ICP, CPD, PDA, NDT, FilterReg, and ICDO algorithms is evaluated on the same set of images used for the evaluation of the proposed PSR algorithm. The percentage of targets matched correctly decreased from 66% to 36% for ICP, from 49% to 34% for CPD, from 58% to 33% for PDA, from 41% to 29% for NDT, from 47% to 43% for FilterReg, and from 47% to 46% for ICDO.
The scenarios where the proposed PSR algorithm performed worse than the ICP, CPD, and PDA algorithms are identified and discussed.

3 UAVs’ Flight Path and Positioning of Images

Images were captured using two UAVs that were operated at Artesia Park, California, USA, as shown in Figure 1. An open field in the park was selected for the operation of the UAVs in order to avoid obstacles such as trees and fences. The field was designated as the fly zone, and a Cartesian coordinate system was defined with an origin set according to the borders of the fly zone as shown in Figure 1.
Fig. 1.
Fig. 1. Fly zone where experiments were performed, Artesia Park, City of Artesia, California, USA.
The UAVs were programmed to fly autonomously along a flight path specified with waypoints. Each UAV takes off from its first waypoint. The first image is captured while hovering at the first waypoint at the desired altitude. The UAV then flies straight toward the second waypoint and hovers momentarily to capture the second image. A feedback loop was included that verified that the UAV reached the waypoint within a margin of error before capturing the image. This process continues until the last waypoint is reached. The UAV lands after capturing the image of the last waypoint.
Both UAVs flew eastbound to cover a rectangular area according to the waypoints shown in Figure 2. One UAV followed waypoints 1 to 4, and the other UAV followed waypoints 5 to 8. Waypoints 1 to 4 correspond to the red area within the fly zone indicated in Figure 1. Waypoints 5 to 8 correspond to the blue area. Figure 2 also shows the location of targets on the ground.
Fig. 2.
Fig. 2. Waypoints of first UAV are points 1, 2, 3, 4, and waypoints of second UAV are points 5, 6, 7, 8.
The UAVs stop and hover at each waypoint to guarantee a required overlap between neighboring images as shown in Figure 3. The overlap between adjacent images is needed because UAVs do not hover at the exact location and direction programmed for each waypoint. Figure 4 shows how the direction of an image is affected by the direction error of the UAV. This direction error is due to the wind and limited accuracy of the UAV’s compass. The ideal direction is the straight line from the start to end points of the flight path as shown in Figure 4. The average wind direction and speed that took place during the measurements are indicated in Figure 1. The wind blew at 6.4 km/h on average in the South-South-East (SSE) direction.
Fig. 3.
Fig. 3. Height of the UAV.
Fig. 4.
Fig. 4. Direction error of the UAV.
In a similar manner, the displacement of the UAV (i.e., traveled distance) is not equal to the ideal displacement. For example, if the next waypoint is 3 m away from the current waypoint, the actual displacement may be of 2.8 m, and the UAV may measure a displacement of 2.9 m. The distribution of the error in displacement was obtained in the field, and it is shown in Figure 5. It shows that the UAV tends to cover a distance shorter than the desired value.
Fig. 5.
Fig. 5. Distribution of the error in UAV’s traveled distance when it flies autonomously along a straight line segment.
A set of images captured by the UAV that covers the area in red in Figure 1 is shown in Figures 6(a) through 6(d). A set of images captured by the UAV that covers the area in blue in Figure 1 is shown in Figures 6(e) through 6(h). Each UAV locates each of its images by calculating the coordinates of the image’s center and corners in the coordinate system. Each UAV also detects and locates targets in each of its images. Figure 7(a) shows the images of Figure 6 after all their centers, corners, and targets have been detected and located. Figure 7(b) and Figure 7(c) show the images of the other two target distributions used for the experimental evaluation in Section 5. It should be noted that the images shown in Figure 2 are perfectly aligned because they were artificially generated via simulation. On the other hand, the images in Figure 6 are aligned according to the GPS and direction measurements of the UAVs, which are not ideal.
Fig. 6.
Fig. 6. Images of the first target distribution used for experimental evaluation.
Fig. 7.
Fig. 7. Overlapped images of target distributions used for experimental evaluation where image centers correspond to UAVs’ GPS coordinates.
The UAVs measure their GPS location at each waypoint. The location of each image center is calculated using the GPS coordinate measured at the corresponding waypoint and the GPS of the origin of the coordinate system indicated in Figure 1. The x and y coordinates of the center of an image correspond to the length of the arc formed by the angle between the GPS coordinates of the waypoint and origin. Several tests were performed experimentally using measuring tape to quantify the length of a test arc. These tests were used to confirm the numerical value of the earth’s radius needed for the calculation of the length of the arc between waypoints. For example, Figure 2 shows that the center of the first image (i.e., first waypoint) is 8 m east and 22 m north of the origin.
The UAVs also measure their direction at each waypoint. The direction corresponds to the direction its nose points to with respect to the North Magnetic Pole, and this direction is measured at every waypoint. The direction of images corresponds to the UAV direction at the corresponding waypoint.
The direction and location of images are then used to determine the coordinates of the image’s corners and targets. The locations of the corners of an image are determined from the coordinate of the center of the image, its direction, and the length of the sides of the images. The length of the sides is determined from the field of view of the camera and the height of the UAV as indicated in Figure 3. This is the reason the images in Figure 6 are rotated as shown in Figure 7. They are affected by errors in direction and displacement of the UAV shown in Figures 4 and 5. Therefore, the actual location and direction of images are not equivalent to the images’ locations and directions shown in Figure 7.
The UAV’s instrumentation needed for these measurements includes GPS unit, altimeter, compass, and camera. The UAVs and camera used were 3DR Solo [1] and Pi Camera [3].
The targets were red cup coasters with a diameter of 20.3 cm, and target detection is performed via detection of the color red and contour of the detected red areas (i.e., targets) using OpenCV [2]. Targets in Figure 6 were made white manually because their original red color made it difficult to see them in the pictures. Target detection is implemented in this article with color detection, but advanced target detection such as deep learning in hyper-spectral [12] or high-resolution [51] images can be integrated with the algorithm proposed in this article. The coordinates of targets correspond to the coordinate of the pixel at the centroid of the red contour. This coordinate was determined using the coordinate values of the pixels at the corners.
It also needs to be noted that an individual target that shows in more than one image does not have the same coordinate in every image due to the deviation of the UAV from its flight path. Therefore, an individual target becomes several different targets even after all images have been located on the coordinate system. For example, a target that shows in four images can be identified as four different targets if the coordinates of the target in each image are different from one another. The centroid of these coordinates can then be used to estimate the location of the target (Section 6).
The algorithm presented in Section 4 has the objective of identifying individual targets that show in more than one image. This target identification is implemented by identifying patterns of targets that show on overlapping images. The problem of identifying and matching target patterns on overlapping images is known in the literature as point set registration, so we refer to the proposed algorithm as the PSR-UAV algorithm.

4 PSR-UAV Algorithm

The PSR-UAV algorithm processes two images at a time, and the algorithm is applied to all possible combinations of two images among all images captured by the UAVs. Therefore, if the UAVs capture N images, the PSR-UAV algorithm first determines what image pairs out of all \(({N \atop 2})\) pairs overlap, and it then proceeds to align the image pairs that overlap. The output of the algorithm is the following. Each image has a set of targets and for each of these targets a list of coordinates is kept. Consider the first target of the first image. If this target is identified in two images including the first one, the list of coordinates of this target will include two entries that correspond to the coordinates of the target in the two images. This list of coordinates is kept for each target in each image, and the centroid of the coordinates of each target is used to estimate the location of the target.
The basic idea behind the PSR-UAV algorithm is illustrated in Figure 8. Consider two images located according to the GPS coordinates and direction recorded by the UAV as shown in Figure 8(a). The first step is to artificially increase the overlapping area between the images as shown in Figure 8(b). Once the overlapping area is estimated, the targets of each picture in the overlapping area are identified. These targets are shown in Figures 8(a) and 8(b). Then, a pattern of targets common to both images is identified. This pattern can be a triangle (i.e., three targets), a line segment (i.e., two targets), or a single target. The second image then goes through a rigid transformation [29] whose translation, rotation, and scaling parameters are estimated from the identified pattern of targets present in both images. Figure 8(c) shows the start and end locations of targets of the second image before and after the rigid transformation, and Figure 8(d) shows the locations of targets of both images after the transformation. Finally, the correspondence of targets in the overlapping area is determined from the Euclidean distance between targets of the original first image and transformed second image as shown in Figure 8(e).
Fig. 8.
Fig. 8. Process of point set registration performed by the PSR-UAV algorithm.

4.1 Finding the Overlap between Two Images

The first step is to increase the overlap between the two images in order to account for the error in UAVs’ displacement and direction. The error can induce image locations in which the images do not overlap even when the images actually overlap. The objective is to identify the targets that can potentially be in both images. For example, Figure 6(g) and Figure 6(h) overlap as shown in Figure 7(a) and Figure 8(a) according to the UAV’s GPS location and direction; however, the actual overlap can be larger due to errors in measurements of GPS location and direction. Therefore, the PSR-UAV algorithm shifts the second image (i.e., Figure 6(h)) toward the first image (i.e., 6(g)) by using the direction vector \(\mathbf {d}_{2,1}\) defined in Equation (1), where \(\mathbf {c}_1\) and \(\mathbf {c}_2\) are the centers of the images and \(\overline{p_\text{dev}}\) is the average displacement error (i.e., average of the distribution of the error in traveled distance shown in Figure 5). Therefore, \(\mathbf {d}_{2,1}\) is a vector from the center of the second to the first image whose length corresponds to the average displacement error.
\begin{equation} \mathbf {d}_{2,1} \doteq \overline{p_\text{dev}} \frac{\mathbf {c}_1 - \mathbf {c}_2}{\Vert \mathbf {c}_1 - \mathbf {c}_2\Vert } \end{equation}
(1)
An example of the overlap calculated by the PSR-UAV algorithm is shown in Figure 8(b). The first image is in green, and it shows the targets of Figure 6(g) that are in the overlapping area calculated by the PSR-UAV algorithm. The second image is in blue, and it shows the targets of Figure 6(h) that are in the overlapping area. These two images correspond to the seventh and eighth images of Figure 7(a), so the location of the first image relative to the second image shown in Figure 8(a) is a result of the GPS coordinates and direction measured by the UAV. When the second image is moved toward the first image in the direction and length of \(\mathbf {d}_{2,1}\) as shown in Figure 8(b), all targets in green are those of the seventh image in Figure 7(a) that fall within the blue perimeter of the shifted second image, and in a similar manner, all targets in blue are those of the eighth image in Figure 7(a) that fall within the green perimeter of the shifted first image. Therefore, when the two images are located according to their GPS location and direction measured by the UAV, not all green targets fall within the blue perimeter of the second image and vice versa as shown in Figure 8(a). The targets shown in Figure 8(a) are the ones considered by the PSR-UAV algorithm to solve the PSR problem.
Two images that are too distant from each other will not overlap even after shifting the second image toward the first image. For example, this is the case of the images in Figure 6(a) and Figure 6(h). Therefore, the PSR-UAV algorithm needs to detect if the images overlap after shifting. This detection is done by the algorithm in Figure 9, which detects overlap using the images’ centers, corners, and edges as shown in Figure 10. The perimeter of the \(i^\text{th}\) image is indicated by coordinates \(\mathbf {p}_{i,j}\), where \(j = 1, 2, \ldots , 8\). These coordinates correspond to the corners and edges’ centers of the image as shown in Figure 10. The center of the \(i^\text{th}\) image is indicated by coordinate \(\mathbf {c}_i\). The algorithm tests each corner and edge center of each image to determine if it is within the other image. This test is shown in Figure 10 for the upper-left corner of the second image (i.e., \(\mathbf {p}_{2,1}\)). In this case, \(\mathbf {p}_{2,1}\) is detected to be within the first image. This detection is performed using vectors \(\mathbf {u}_j\) and \(\mathbf {v}_j\) shown in Figure 10, where \(j = 1, 2, 3, 4\), and the detection is performed on \(\mathbf {p}_{2,1}\), \(\mathbf {p}_{2,2}\), ..., \(\mathbf {p}_{2,8}\). Point \(\mathbf {p}_{2,j}\) is within the first image if \(\mathbf {u}_1 \cdot \mathbf {v}_1\), ..., \(\mathbf {u}_4 \cdot \mathbf {v}_4\) are all positive, meaning that \(\mathbf {u}_j\) points toward the inside of the first image. The algorithm decides that the images overlap if any of the points \(\mathbf {p}_{2,1}\), \(\mathbf {p}_{2,2}\), ..., \(\mathbf {p}_{2,8}\) are detected inside the first image. The complete algorithm is shown in Figure 9. It is executed twice for every pair of images. In the first run, the corners and edge centers of the second image are tested as shown in Figure 10, and in the second run, the corners and edge centers of the first image are tested.
Fig. 9.
Fig. 9. TestOverlap algorithm that determines if two images overlap.
Fig. 10.
Fig. 10. Operation of the TestOverlap algorithm (Figure 9) that determines if two images overlap.

4.2 Numbering Targets of Two Overlapping Images

The targets of two overlapping images are first numbered in order of proximity to the center of the other overlapping image. For example, consider the second and third images in Figure 7(b), which correspond to the first and second image within the PSR-UAV algorithm. Let \(\mathbf {p}_{i,j}\) be the \(j^\text{th}\) target of the \(i^\text{th}\) image. Figure 11 shows the targets of the second image (i.e., targets of the third image in Figure 7(b)). Let \(\mathbf {d}_{2,1}\) be the unit-length vector that points from the center of the second image to the center of the first image as shown in Figure 11, and let \(p_{2,j} \doteq \mathbf {p}_{2,j} \cdot \frac{\mathbf {d}_{2,1}}{|\mathbf {d}_{2,1}|}\) be the projection of the \(j^\text{th}\) target of the second image, \(\mathbf {p}_{2,1}\). Figure 11 shows the projections of the first three targets. Targets of the second image are ordered in descending order of projections, so \(p_{2,1}\) is the largest projection among all target projections on \(\mathbf {d}_{2,1}\) as shown in Figure 11.
Fig. 11.
Fig. 11. Ordering of targets of the second image from closest to farthest to the center of the first image.
Targets of the first image are numbered in order of proximity to the center of the second image in the same manner.

4.3 Matching Target Patterns in Overlapping Areas: Triangles, Line Segments, Points

The next step is to identify equivalent target patterns on the first and second images. Three patterns are considered: triangles, line segments, and points. First, the PSR-UAV algorithm attempts to find matching triangles. If this test fails, the algorithm attempts to find matching line segments, and if this test fails, the algorithm attempts to find matching points. If matching points are not found, the algorithm determines that the overlapping images do not have any common targets. On the other hand, matching patterns are an indication of common targets in the overlapping region of the images.

4.3.1 Matching Triangles.

A set of triangles is found for each image as shown in Figure 12. Let \(N_1\) and \(N_2\) be the number of overlapping targets of the first and second images, respectively. These targets are shown in white in Figure 12. Let \(\lbrace \bigtriangleup _{i,1},\bigtriangleup _{i,2},....,\bigtriangleup _{i,N_i-2}\rbrace\) be the set of triangles of the \(i^\text{th}\) image, where \(i = 1,2\). \(\bigtriangleup _{i,1}\) includes targets \(\mathbf {p}_{i,1}\), \(\mathbf {p}_{i,2}\), and \(\mathbf {p}_{i,3}\); \(\bigtriangleup _{i,2}\) includes targets \(\mathbf {p}_{i,2}\), \(\mathbf {p}_{i,3}\), and \(\mathbf {p}_{i,4}\); and so on. The last triangle, \(\bigtriangleup _{i,N_i-2}\), includes targets \(\mathbf {p}_{i,N_i-2}\), \(\mathbf {p}_{i,N_i-1}\), and \(\mathbf {p}_{i,N_i}\). Two triangles are said to match if they are sufficiently similar in the following manner. Let the deviation between two triangles, denoted by \(\bigtriangleup _\text{dev}\), be a metric of their difference. Two triangles match if their deviation \(\bigtriangleup _\text{dev}\) is below a threshold measured experimentally. Let \(\bigtriangleup _\text{dev}\) be defined as follows. Consider any two triangles \(\bigtriangleup _{1,n_1}\) and \(\bigtriangleup _{2,n_2}\), where \(n_i = 1,2,\ldots ,N_i-2\), such as the two yellow triangles in Figure 12. Let \(\lbrace l_{i,n_i,1},l_{i,n_i,2},l_{i,n_i,3}\rbrace\) be the length of the edges, from shortest to largest, of \(\bigtriangleup _{i,n_i}\); let \(\mathbf {b}_{i,n_i}\) be the angle bisector of the smallest angle of \(\bigtriangleup _{i,n_i}\); and let \(\mathbf {\Delta }_{i,n_i,j}\) be the coordinate of the \(j^\text{th}\) vertex of \(\bigtriangleup _{i,n_i}\). For example, the vertices of the first triangle of the \(i^\text{th}\) image (i.e., \(\bigtriangleup _{i,1}\)) are \(\mathbf {p}_{i,1}\), \(\mathbf {p}_{i,2}\), and \(\mathbf {p}_{i,3}\), so \(\mathbf {\Delta }_{i,1,1} = \mathbf {p}_{i,1}\), \(\mathbf {\Delta }_{i,1,2} = \mathbf {p}_{i,2}\), and \(\mathbf {\Delta }_{i,1,3} = \mathbf {p}_{i,3}\). The deviation between triangles \(\bigtriangleup _{1,n_1}\) and \(\bigtriangleup _{2,n_2}\) is given in Equation (2) as a function of the centroid deviation, length deviation, and angle deviation between the triangles. The centroid deviation, denoted by \(c_\text{dev}\), is the Euclidean distance between the centroids of \(\bigtriangleup _{1,n_1}\) and \(\bigtriangleup _{2,n_2}\) as defined in Equation (3), where \(\frac{1}{3}\sum _{j=1}^{3}\mathbf {\Delta }_{i,n_i,j}\) is the centroid of \(\bigtriangleup _{i,n_i}\). The length deviation, denoted by \(l_\text{dev}\), is the root mean square (RMS) error between the lengths of the triangles’ edges, from shortest to largest, as defined in Equation (4). The angle deviation, denoted by \(\alpha _\text{dev}\), is the angle between the bisectors of the smallest angles of the triangles, and it is defined in Equation (5), where \(\mathbf {b}_{i,n_i}\) is the bisector of the smallest angle of \(\bigtriangleup _{i,n_i}\). Figure 12 shows the bisectors of two matching triangles that are indicated in yellow.
\begin{equation} \bigtriangleup _\text{dev}(\bigtriangleup _{1,n_1},\bigtriangleup _{2,n_2}) \doteq \sqrt {\left(\frac{c_\text{dev}}{\overline{c_\text{dev}} + \sigma _{c_\text{dev}}}\right)^2 + \left(\frac{l_\text{dev}}{\overline{l_\text{dev}} + \sigma _{l_\text{dev}}}\right)^2 + \left(\frac{\alpha _\text{dev}}{\overline{\alpha _\text{dev}} + \sigma _{\alpha _\text{dev}}}\right)^2} \end{equation}
(2)
\begin{equation} c_\text{dev}(\bigtriangleup _{1,n_1},\bigtriangleup _{2,n_2}) \doteq \frac{1}{3}\left\Vert \sum _{i=1}^{3}\mathbf {\Delta }_{1,n_1,i} - \mathbf {\Delta }_{2,n_2,i}\right\Vert \end{equation}
(3)
\begin{equation} l_\text{dev}(\bigtriangleup _{1,n_1},\bigtriangleup _{2,n_2}) \doteq \sqrt {\sum _{i=1}^3 (l_{1,n_1,i}-l_{2,n_2,i})^2} \end{equation}
(4)
\begin{equation} \alpha _\text{dev}(\bigtriangleup _{1,n_1},\bigtriangleup _{2,n_2}) \doteq {|\angle (\mathbf {b}_{1,n_1},\mathbf {b}_{2,n_2})|} \end{equation}
(5)
Fig. 12.
Fig. 12. Triangles of the first and second images that are compared with each other to find the pair of triangles of minimum deviation shown in yellow.
Deviations \(c_\text{dev}\), \(l_\text{dev}\), \(\alpha _\text{dev}\), and \(\bigtriangleup _\text{dev}\) cannot be negative by definition, and they are all zero for perfectly located and aligned images, which is not true in practice. Therefore, the deviation metrics take on positive and random values whose distributions are reported in Figure 13. The distributions were obtained experimentally as follows. Consider three targets that show on two images such as those of the yellow triangles in Figure 12. Now, consider the deviation between the two triangles. This deviation is calculated for every two same-target triangles for every two images collected for the experimental evaluation described in Section 5. Figure 13(a) and Figure 13(b) show the distribution of \(l_\text{dev}\) and \(\alpha _\text{dev}\). The distribution of \(c_\text{dev}\) is approximated to the distribution of the error in displacement shown in Figure 5 (i.e., the error in UAV’s direction is disregarded). The mean and standard deviation of these distributions are listed in Table 1, and their values are used to normalize each deviation metric so that \(\bigtriangleup _\text{dev}\) can be defined as in Equation (2), which is the RMS value of the normalized deviation metrics. Figure 13(c) shows the distribution of \(\bigtriangleup _\text{dev}\).
Fig. 13.
Fig. 13. Distributions of the deviation metrics of triangles.
Table 1.
  Deviation Metric
\(\overline{c_\text{dev}}\), \(\sigma _{c_\text{dev}}\)\(\overline{\alpha _\text{dev}}\), \(\sigma _{\alpha _\text{dev}}\)\(\overline{l_\text{dev}}\), \(\sigma _{l_\text{dev}}\)\(\overline{\bigtriangleup _\text{dev}}\), \(\sigma _{\bigtriangleup _\text{dev}}\)\(\overline{S_\text{dev}}\), \(\sigma _{S_\text{dev}}\)\(\overline{p_\text{dev}}\), \(\sigma _{p_\text{dev}}\)
(Meters)(Degrees)(Meters)(-)(-)(Meters)
 Triangle0.51, 0.443.84, 3.180.04, 0.031.09,1.04N/AN/A
PatternLine Segment0.51, 0.443.87, 3.290.02, 0.02N/A1.00, 1.15N/A
 PointN/AN/AN/AN/AN/A0.51, 0.44
Table 1. Mean and Standard Deviation of Deviation Metrics
Finally, triangles \(\bigtriangleup _{1,n_1}\) and \(\bigtriangleup _{2,n_2}\) match if their deviation \(\bigtriangleup _\text{dev}\) is below the threshold defined in Equation (6), which depends on the mean and standard deviation of the distribution of \(\bigtriangleup _\text{dev}\) (Figure 13(c)) and constant k, which was determined experimentally (\(k = 1.55\)).
\begin{equation} \bigtriangleup _\text{th} \doteq \overline{\bigtriangleup _\text{dev}} + k\sigma _{\bigtriangleup _\text{dev}} \end{equation}
(6)
The calculation of \(\alpha _\text{dev}\) requires an extra step not mentioned previously for clarity. The reason is that \(\alpha _\text{dev}\) depends on identifying the minimum angle of triangles \(\bigtriangleup _{1,n_1}\) and \(\bigtriangleup _{2,n_2}\), and the minimum angle can be misidentified due to tilting of the UAV when capturing an image while hovering in the presence of wind. Consider any three targets covered in both images as shown in Figure 14. \(\bigtriangleup _{1,n_1}\) is shown in green, and \(\bigtriangleup _{2,n_2}\) is shown in blue. Assume that \(\bigtriangleup _{1,n_1}\) and \(\bigtriangleup _{2,n_2}\) correspond to the same targets on the ground and that they do not align in Figure 14(a) due to the error in direction and displacement of the UAV. Therefore, \(\bigtriangleup _{1,n_1}\) and \(\bigtriangleup _{2,n_2}\) are not on top of one another in Figure 14(a). The angles of \(\bigtriangleup _{1,n_1}\) and \(\bigtriangleup _{2,n_2}\) in Figures 14 and 15 are named after their relative values, so \(\alpha _\text{s}\), \(\alpha _\text{m}\), and \(\alpha _\text{l}\) are the smallest, medium, and largest angles respectively. The smallest angle of \(\bigtriangleup _{1,n_1}\) and \(\bigtriangleup _{2,n_2}\) is identified correctly in Figure 14, and it is not in Figure 15. The smallest angle of \(\bigtriangleup _{2,n_2}\) is mistaken for the medium angle in Figure 15(a), which can be observed by comparing \(\alpha _\text{s}\) and \(\alpha _\text{m}\) of color blue in Figure 14(a) and Figure 15(a). The tilted UAV causes a change of perspective in the image that causes the angles to change. In this case, \(\alpha _\text{s}\) of \(\bigtriangleup _{2,n_2}\) increases its value, and \(\alpha _\text{m}\) decreases its value, becoming the smallest angle of \(\bigtriangleup _{2,n_2}\).
Fig. 14.
Fig. 14. Alignment of triangles that shows correct order of angles on the triangle of the second image.
Fig. 15.
Fig. 15. Alignment of triangles that shows incorrect order of angles on the triangle of the second image.
The following test is performed to detect misidentification of the smallest angle. \(\bigtriangleup _{2,n_2}\) is moved toward \(\bigtriangleup _{1,n_1}\) to match the vertex of the largest angle as shown in Figure 14(b) and Figure 15(b). \(\bigtriangleup _{2,n_2}\) is then rotated along this vertex until the edge that joins the vertices of \(\alpha _\text{s}\) and \(\alpha _\text{l}\) are colinear. This rotation is shown in Figure 14(c) and Figure 15(c). There can be two outcomes: (1) \(\bigtriangleup _{1,n_1}\) and \(\bigtriangleup _{2,n_2}\) overlap as shown in Figure 14(c); (2) \(\bigtriangleup _{1,n_1}\) and \(\bigtriangleup _{2,n_2}\) do not overlap as shown in Figure 15(c). Overlapping triangles as in Figure 14(c) is indication that \(\alpha _\text{s}\) is identified correctly in both \(\bigtriangleup _{1,n_1}\) and \(\bigtriangleup _{2,n_2}\). Non-overlapping triangles as in Figure 15(c) is indication that \(\alpha _\text{s}\) was misidentified in one of the triangles. The solution is to swap the vertices of \(\alpha _\text{s}\) and \(\alpha _\text{m}\) in \(\bigtriangleup _{2,n_2}\), so the vertex of \(\alpha _\text{m}\) is considered the vertex of the smallest angle, and the vertex of \(\alpha _\text{s}\) is considered the vertex of the medium angle. Then, the same rotation is performed again, and if the triangles do not overlap once more, the vertices are reordered in the original sequence from the smallest angle to largest angle, and the vertices of \(\alpha _\text{m}\) and \(\alpha _\text{l}\) are swapped. The rotation is performed one last time, and if the triangles still do not overlap, the vertices are reordered in the original sequence from the smallest angle to largest angle again, and the vertices of \(\alpha _\text{s}\) and \(\alpha _\text{l}\) are swapped. The bisector is determined on the first vertex of the reordered list of angles.
The deviation among triangles is calculated in the following order: \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,1},\bigtriangleup _{2,1})\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,1},\) \(\bigtriangleup _{2,2})\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,2},\bigtriangleup _{2,1})\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,1},\bigtriangleup _{2,3})\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,3},\bigtriangleup _{2,1})\), \(\ldots\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,2},\bigtriangleup _{2,2})\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,2},\bigtriangleup _{2,3})\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,3},\bigtriangleup _{2,2})\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,2},\bigtriangleup _{2,4})\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,4},\bigtriangleup _{2,2})\), \(\ldots\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,N_1-3},\bigtriangleup _{2,N_2-3})\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,N_1-3},\bigtriangleup _{2,N_2-2})\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,N_1-2},\bigtriangleup _{2,N_2-3})\), \(\bigtriangleup _\text{dev}(\bigtriangleup _{1,N_1-2},\bigtriangleup _{2,N_2-2})\), and the first deviation to be found below the threshold \(\bigtriangleup _\text{th}\) is used to perform point set registration on the two images as will be explained in Section 4.3.4. The reason for this order is that the first triangle of an image is the closest triangle to the center of the other image, and the closer it is, the higher the probability that the triangle is covered by both images. Therefore, the first comparison is between the first triangle of the first image and the first triangle of the second image (i.e., \(\bigtriangleup _{1,1}\) and \(\bigtriangleup _{2,1}\)). The second comparison is between the first triangle of the first image and the second triangle of the second image (i.e., \(\bigtriangleup _{1,1}\) and \(\bigtriangleup _{2,2}\)). In the third comparison, the images alternate, so the comparison is between the first triangle of the second image and the second triangle of the first image (i.e., \(\bigtriangleup _{1,2}\) and \(\bigtriangleup _{2,1}\)). The images alternate again in the fourth comparison, so the comparison is between the first triangle of the first image and the third triangle of the second image (i.e., \(\bigtriangleup _{1,1}\) and \(\bigtriangleup _{2,3}\)). The images alternate once again in the fifth comparison, so the comparison is between the first triangle of the second image and the third triangle of the first image (i.e, \(\bigtriangleup _{1,3}\) and \(\bigtriangleup _{2,1}\)), and so on.
The PSR-UAV algorithm switches to finding matching segments if all deviations among triangles are above \(\bigtriangleup _\text{th}\).

4.3.2 Matching Segments.

Let \(\lbrace S_{i,1}, S_{i,2}, \dots , S_{i,N_i-1}\rbrace\) be the set of segments of the \(i^\text{th}\) image, where \(i = 1, 2\). \(S_{i,1}\) is the line segment that joins targets \(\mathbf {p}_{i,1}\) and \(\mathbf {p}_{i,2}\); \(S_{i,2}\) is the line segment that joins targets \(\mathbf {p}_{i,2}\) and \(\mathbf {p}_{i,3}\); and so on. The last segment, \(S_{i,N_i-1}\), is the line segment that joins targets \(\mathbf {p}_{i,N_i-1}\) and \(\mathbf {p}_{i,N_i}\). The deviation between two segments, denoted by \(S_\text{dev}\), is defined in the same manner as the deviation between triangles in Equation (2). The centroid deviation for segments is defined in Equation (7), where \(\mathbf {s}_{i,n_i,j}\) is the coordinate of the \(j^\text{th}\) end of \(\mathbf {s}_{i,n_i}\). For example, the two ends of the first segment of the \(i^\text{th}\) image (i.e., \(\mathbf {s}_{i,1}\)) are \(\mathbf {s}_{i,1,1} = \mathbf {p}_{i,1}\) and \(\mathbf {s}_{i,1,2} = \mathbf {p}_{i,2}\), and the two ends of the second segment of the \(i^\text{th}\) image (i.e., \(\mathbf {s}_{i,2}\)) are \(\mathbf {s}_{i,2,1} = \mathbf {p}_{i,2}\) and \(\mathbf {s}_{i,2,2} = \mathbf {p}_{i,3}\), and so on. The length deviation is defined in Equation (8) as the absolute value of the difference of segment lengths. The angle deviation is defined in Equation (9). It is the smallest angle between the two lines that correspond to the segments.
\begin{equation} c_\text{dev}(S_{1,n_1},S_{2,n_2}) \doteq {\frac{1}{2}\left\Vert \sum _{j=1}^{2}\mathbf {s}_{1,n_1,j} - \mathbf {s}_{2,n_2,j}\right\Vert } \end{equation}
(7)
\begin{equation} l_\text{dev}(S_{1,n_1},S_{2,n_2}) \doteq |\Vert \mathbf {s}_{1,n_1,2}-\mathbf {s}_{1,n_1,1}\Vert -\Vert \mathbf {s}_{2,n_2,2}-\mathbf {s}_{2,n_2,1}\Vert | \end{equation}
(8)
\begin{equation} \alpha _\text{dev}(S_{1,n_1},S_{2,n_2}) \doteq {|\angle (\mathbf {s}_{1,n_1,2}-\mathbf {s}_{1,n_1,1},\mathbf {s}_{2,n_2,2}-\mathbf {s}_{2,n_2,1})|} \end{equation}
(9)
The distributions of the deviation metrics are shown in Figure 16. They were found experimentally in the same way that they were found for triangles as described in Section 4.3.1. Figures 16(a) through 16(c) show the distribution of \(l_\text{dev}\), \(\alpha _\text{dev}\), and \(S_\text{dev}\), respectively. The distribution of \(c_\text{dev}\) is approximated to the distribution of the error in displacement, which is shown in Figure 5 (i.e., the error in UAV’s direction is disregarded). The mean and standard deviation of these distributions are listed in Table 1, and their values are used to normalize each deviation metric so that \(S_\text{dev}\) can be defined in the same way of \(\bigtriangleup _\text{dev}\) in Equation (2).
Fig. 16.
Fig. 16. Distributions of the deviation metrics of line segments.
\(S_{1,n_1}\) and \(S_{2,n_2}\) match if their deviation, \(S_\text{dev}(S_{1,n_1},S_{2,n_2})\), is below the threshold \(S_\text{th}\) given in Equation (10), which depends on the mean and standard deviation of the distribution of \(S_\text{dev}\) shown in Figure 16(c).
\begin{equation} S_\text{th} \doteq \overline{S_\text{dev}} + k \sigma _{S_\text{dev}} \end{equation}
(10)
The deviation among segments is calculated in the same order of triangles, and the first deviation to be found below the threshold \(S_\text{th}\) is used to perform point set registration on the two images as will be explained in Section 4.3.4. The PSR-UAV algorithm switches to finding matching points if all deviations among segments are above \(S_\text{th}\).

4.3.3 Matching Points.

A point corresponds to the coordinate of a target, so there are \(N_1\) and \(N_2\) points in the first and second images, respectively. Let \(\mathbf {p}_{i,n_i}\) be the \(n_i^\text{th}\) point in the \(i^\text{th}\) image. The deviation between \(\mathbf {p}_{1,n_1}\) and \(\mathbf {p}_{2,n_2}\) is defined in Equation (11) as the Euclidean distance between the points. The distribution of \(p_\text{dev}\) is approximated to the distribution of the error in displacement (Figure 5) (i.e., the error in UAV’s direction is disregarded). The mean and standard deviation of this distribution are listed in Table 1, and their values are used to determine the threshold, \(p_\text{th}\), that indicates when two points match. The threshold is given in Equation (12).
\begin{equation} p_\text{dev}(\mathbf {p}_{1,n_1},\mathbf {p}_{2,n_2}) \doteq \Vert \mathbf {p}_{1,n_1} - \mathbf {p}_{2,n_2}\Vert \end{equation}
(11)
\begin{equation} p_\text{th} \doteq \overline{p_\text{dev}} + k\sigma _{p_\text{dev}} \end{equation}
(12)
The deviation among points is calculated in the same order of triangles, and the first deviation to be found below the threshold \(p_\text{th}\) is used to perform point set registration on the two images as will be explained in Section 4.3.4. The PSR-UAV algorithm decides that the two images have no targets in common if all deviations among points are above \(p_\text{th}\).

4.3.4 Point Set Registration.

If a matching target pattern (i.e., triangle, line segment, point) is identified, the next step in the PSR-UAV algorithm is to perform a rigid transformation (i.e., translation, rotation, and scaling) on the second image so that the correspondence between targets of the two images can be established.
The location of the second image is corrected by moving the image (i.e., translation) by the length and direction of the vector that starts at the centroid of the matching pattern on the second image and ends at the centroid of the matching pattern on the first image. For example, consider the image pair shown in Figure 12. The matching target pattern is the yellow triangles. Then, translation of the second image takes place by moving it toward the first image until the centroids of the yellow triangles match. The orientation and scale of the second image are also corrected if the matching patterns are triangles or line segments. The orientation is corrected by rotating the second image around the centroid of its matching pattern. The angle of rotation is the angle deviation of the matching patterns (i.e., \(\alpha _\text{dev}\)), and the direction of rotation is such that \(\alpha _\text{dev}\) is zero after rotation. For example, the second image of Figure 12 is rotated until the angle bisectors of the yellow triangles (i.e., vectors \({\bf b_{1,1}}\) and \({\bf b_{2,11}}\)) are colinear. The image is then scaled so that the mean square error of the distance between corresponding targets of the matching patterns is minimized. In Figure 12, it is the mean square error of the distance between \({\bf p_{1,2}}\) and \({\bf p_{2,12}}\), \({\bf p_{1,1}}\) and \({\bf p_{2,13}}\), and \({\bf p_{1,3}}\) and \({\bf p_{2,11}}\) after the second image is translated and rotated.
The next step is to determine the correspondence between targets. Let a target pair be defined as a set of two targets, one target from the first image and the other target from the second image. Let the distance of a target pair be the Euclidean distance between the targets of the pair. The target pair of shortest distance is found first. The next target pair of shortest distance is then found, omitting the pair previously found. This process continues until the distance is greater than the threshold \(T_\text{th}\) defined in Equation (13), where \(\overline{l_\text{dev}}\) and \(\sigma _{l_\text{dev}}\) are the mean and standard deviation of the length deviation of line segments. The distribution of this deviation is shown in Figure 16(a).
\begin{equation} T_\text{th} = \frac{\overline{l_\text{dev}}}{2}+k\frac{\sigma _{l_\text{dev}}}{2} \end{equation}
(13)
Finally, target pairs of distance less than \(T_\text{th}\) are considered to be one target only on the ground, and we refer to them as matched target pairs. Target pairs of distance greater than \(T_\text{th}\) are considered to be two different targets.

5 Experimental Evaluation

There were two UAVs that flew at an altitude of 4.1 m on a straight path eastbound as indicated in Figure 1. Experiments demonstrated that all targets were detected in the images as long as they were taken at altitudes not greater than 5.8 m. The reason for selecting an altitude of 4.1 m is that it is 1.7 m below the maximum altitude for reliable target detection. Therefore, reliable target detection could be performed as long as the error in altitude is not greater than 1.7 m, which is highly unlikely. The maximum altitude tested was 25 m, and only 65% of the targets were detected at this altitude.
The horizontal and vertical fields of view of images covered a distance of 5 m and 3.75 m, respectively, when the UAVs were 4.1 m high. The distance between the UAVs was 2.5 m, which was sufficient to avoid collisions according to the distribution of distance error reported in Figure 5. Therefore, the vertical overlap between images of the two UAVs indicated in Figure 1 is 1.25 m. The UAVs stopped and hovered to capture images every 3 m as indicated in Figure 3, so the horizontal overlap between images of the same UAV indicated in Figure 3 was 2 m. The overlap of 2 m guarantees that the actual overlap after considering the maximum displacement error of 1.2 m reported in Figure 5 is 0.8 m at least. This minimum overlap between adjacent images guarantees that each target is covered by one image at least. The overlap of 2 m takes place on the left and right sides of the image as shown in Figure 3, so the distance at the center of the image that does not overlap is 1 m long. This length was chosen to maximize the overlapping area while achieving a high probability that an image has a non-overlapping area as well given that the maximum displacement error is 1.2 m. Maximizing the overlapping area allows the evaluation of the PSR-UAV algorithm under worst-case scenarios because the probability of matching incorrect target patterns increases when there are more targets to be considered for potential good matches.
Each UAV captured four images, so each UAV traveled a distance of 9 m total because the first image captured was at the starting point of the flight. Targets were placed within the overlapping area indicated in Figure 1 on an area of size 10 m by 1.25 m. This area along with the targets can be seen in Figure 7.
The UAVs flew over nine different target distributions, which are divided into three groups. The first group was generated as follows. First, a total of 80 targets were manually placed on the ground in the overlapping area. The location of targets within this area was random as shown in Figure 6. The target density of this target distribution is \(\frac{80}{10 \times 1.25} = 6.4\) targets/\(\text{m}^2\). Second, 20 of the 80 targets were randomly selected and removed to generate a second target distribution of density \(\frac{80-20}{10 \times 1.25} = 4.8\) targets/\(\text{m}^2\), and third, another 20 targets were randomly selected and removed to generate a third target distribution of density \(\frac{80-20-20}{10 \times 1.25} = 3.2\) targets/\(\text{m}^2\). The second group of distributions was generated in the same way but from a different initial distribution of 80 targets. The third group of target distributions was artificially generated via simulation, not on the field. For this group, the same location and orientation of the images of the second group were used, which can be observed by comparing Figure 7(b) and Figure 7(c), with a new random distribution of 80 targets. The three distributions of 80 targets are shown in Figure 7.
The reason for generating the third group of target distributions via simulation is that target locations can then be determined without error. For example, the target locations presented in Figure 2 are without error, which are later used in Section 6 to compare them with the target locations estimated by the PSR-UAV algorithm.
Another aspect to be considered for reliable target detection at different UAV altitudes is that the distance between the borders of two adjacent targets on the ground cannot be less than a minimum. Otherwise, the target detection process detects these two targets as one single target. The reason for this problem is that target detection is performed using the contours of red areas in the image (i.e., contours of the red cup coasters used as targets), and the contours overlap one another when the targets are too close to each other on the ground. The minimum distance between the borders of adjacent targets for reliable detection was 25 cm at an altitude of 4.1 m. The maximum altitude tested was 25 m, and the minimum distance increased to 57 cm. Therefore, the minimum distance between adjacent targets in the nine target distributions used for the experimental evaluation was always greater than 25 cm.
The PSR-UAV algorithm was written in Python 2.7 and run on a Windows 10 computer with an Intel(R) Core(TM) i7-8700 CPU at 3.2 GHz and RAM of 16 GB. The average execution time of the PSR-UAV algorithm on two images was 29.4 ms. The algorithm was run offline. Images along with their corresponding GPS location and direction were logged during the UAV flights. The PSR-UAV algorithm was run later in the laboratory.

5.1 Target Matching Rate

The goal of the experimental evaluation is to determine the effectiveness of the PSR-UAV algorithm in matching targets that show in two images that overlap. This effectiveness is quantified using the target matching rate(TMR) defined as follows. Consider two images that overlap. Let the number of targets on the ground that are within the overlapping area determined by the PSR-UAV algorithm be N. For example, in Figure 8(a), there are 17 green targets on the first image and 13 blue targets on the second image. These images have 13 targets in common because all targets of the second image are covered in the first image, which can be confirmed visually on Figure 6(g) and Figure 6(h). Figure 6(g) corresponds to the first image (i.e., green image in Figure 8(a)), and Figure 6(h) corresponds to the second image (i.e., blue image in Figure 8(a)). Therefore, the total number of targets in the overlapping area determined by the PSR-UAV algorithm is \(N = 17\), which is the result of 13 targets common to both images plus 4 targets that are present on the first image only. A PSR algorithm performs correctly if targets that are present in one image only are not matched to targets of the other image and targets that are present in both images are matched with the correct target of the other image. Let n be the number of these targets that are correctly registered by the PSR algorithm. The TMR is \(\frac{n}{N}\times 100\), so it is the percentage of targets present in the overlapping area determined by the PSR-UAV algorithm that are registered correctly.
The TMR was calculated for 11 different algorithms: PSR-UAV, GPS-only, PSR-ICP, ICDO, FilterReg, PSR-probabilistic, PSR-UAV-rigid, PSR-rigid, PSR-NDT, PSR-UAV-affine, and PSR-affine. The GPS-only algorithm does not do any modifications to the original images captured by the UAVs. It locates them and directs them according to the GPS location and direction measured by the UAVs as shown in Figure 7, and it then finds matched target pairs for every overlapping pair of images as explained in Section 4.3.4. PSR-ICP was proposed in [8] (i.e., ICP in Section 2); ICDO was proposed in [42]; FilterReg was proposed in [16]; PSR-probabilistic was proposed in [5] (i.e., PDA in Section 2); PSR-rigid and PSR-affine were proposed in [29] (i.e., CPD in Section 2), and PSR-NDT was proposed in [39] (i.e., NDT in Section 2). PSR-UAV-rigid and PSR-UAV-affine are the result of adjusting target locations of the second image with our PSR-UAV algorithm first followed by adjusting the targets again with PSR-rigid or PSR-affine, which were proposed in [29]. The idea behind cascading our PSR-UAV algorithm with the PSR-rigid algorithm of [29] (i.e., PSR-UAV-rigid) and with the PSR-affine algorithm of [29] (i.e., PSR-UAV-affine) was to test if an incremental improvement on TMR could be made. We used the implementation of the PSR-ICP and PSR-NDT algorithms available in the Computer Vision Toolbox of Matlab [27]. The Extrapolate option was set for PSR-ICP. We used the implementation of ICDO that its authors made publicly available. ICDO is a learning-based algorithm that requires training. We trained the algorithm for each of the nine target distributions. There are eight images for each distribution as shown in Figure 7. Registration took place by setting the first and second images registered by our algorithm, PSR-UAV, as the scene and model images of ICDO. FilterReg is also publicly available, and we evaluated it using their rigid-pt2pl script adapted to register our first and second images with the same configuration of the script. For PSR-NDT, the gridStep option was set to 3 m, which is the smallest length that reached the minimum required number of targets in the area that voxelizes all images. We used the Matlab implementation of PSR-CPD published in [29] using the rigid and affine options. We used the C++ implementation of the PSR-probabilistic algorithm published in [5] and made publicly available in [4]. The radius parameter of this algorithm was set to 73 cm, which corresponds to \(\overline{p_\text{dev}} + \frac{1}{2}\sigma _{p_\text{dev}}\). The \(\frac{1}{2}\) factor was found by trial and error. The performance of the algorithm decreased for factors above and below \(\frac{1}{2}\).
Figure 8(a) shows the case of the overlapping images shown in Figure 6(g) and Figure 6(h) for GPS-only. Figure 17(b) shows the locations of the targets of the second image before and after the PSR-ICP algorithm. The initial target locations correspond to those determined from the GPS location and direction of the second image measured by the UAV. Therefore, the initial locations are equivalent to those of GPS-only shown in Figure 8(a). The new locations of targets of the second image determined by PSR-ICP are used to identify matched target pairs, which are indicated with a solid line joining the matched targets. These matched target pairs are also determined as explained in Section 4.3.4. In a similar way, Figures 17(a) through (h) show the locations of the targets of the second image before and after the PSR-UAV, PSR-ICP [8], PSR-probabilistic [5], PSR-UAV-rigid, PSR-rigid [29], PSR-NDT [39], PSR-UAV-affine, and PSR-affine [29] algorithms, respectively, along with the matched target pairs. Not all algorithms were included in Figure 17 for brevity. A visual comparison of Figure 17(a) to Figure 17(h) shows that our algorithm, PSR-UAV, matches targets more effectively than all other algorithms. In fact, Figure 17(a) shows that PSR-UAV matches all targets correctly, so its TMR in this case is 100%. The TMR of GPS-only is 23% because 4 out of the 17 targets are registered correctly. These 4 targets are the 4 green targets most to the left in Figure 8(a) because they do not correspond to any of the blue targets, and they were not matched to any blue target. All other green targets have a matching blue target, but none of them are matched correctly by GPS-only. The TMR of PSR-ICP is 83% because it registers 14 of the 17 targets correctly. The TMR of PSR-probabilistic is 53% because it registers 9 of the 17 targets correctly. The TMR PSR-NDT is 35% because it registers 6 of the 17 targets correctly. The TMR of PSR-UAV-rigid, PSR-rigid, PSR-UAV-affine, and PSR-affine is 29% because they are able to register 5 of the 17 targets correctly.
Fig. 17.
Fig. 17. Selected case of superior target matching rate for the PSR-UAV algorithm.
The images in Figure 17 were handpicked to illustrate an example in which our PSR-UAV algorithm achieves a TMR of 100% while all other algorithms do not, but this is not always the case. For example, Figure 18 to Figure 21 show cases when PSR-ICP, PSR-probabilistic, PSR-rigid, and PSR-affine outperform our PSR-UAV algorithm. PSR-NDT never outperformed PSR-UAV. However, even though the PSR-UAV does not always outperform all algorithms, on average, our PSR-UAV algorithm does achieve a higher TMR as explained next.
Fig. 18.
Fig. 18. Selected case of superior target matching rate for the PSR-ICP algorithm.
The same comparison of Figure 17 was performed for every pair of overlapping images for all target distributions. For every pair, the TMR of each of the 11 PSR algorithms was determined. Figures 22(a) through 22(c) show the average TMR as a function of target density for comparison, and they correspond to the first, second, and third groups of target distributions, respectively. These results show that PSR-UAV outperforms all other algorithms on average. The TMR of the PSR-UAV is maximum at 87% in Figure 22(a), 82% in Figure 22(b), and 86% in Figure 22(c) for the minimum target density of 3.2 targets/\(\text{m}^2\); the TMR of PSR-UAV decreases as the target density increases. The TMR is minimum at 63% in Figure 22(a), 48% in Figure 22(b), and 80% in Figure 22(c) for the maximum target density of 6.4 targets/\(\text{m}^2\). The next algorithms to perform best are GPS-only and PSR-ICP. Their TMR slightly decreases with target density. For GPS-only, it decreases from 49% to 44% in Figure 22(a), from 55% to 49% in Figure 22(b), and from 54% to 53% in Figure 22(c). The TMR of PSR-ICP is similar to the TMR of GPS-only, and all other algorithms do not show significant improvement over GPS-only.

5.2 Image Matching Rate

Figures 22(d) through 22(f) show the average image matching rate(IMR) of the first, second, and third groups of target distributions, respectively. The IMR is defined as the percentage of image pairs matched correctly by a PSR algorithm. A pair of overlapping images is considered to have been matched correctly if all target pairs are matched correctly after applying the steps explained in Section 4.3.4 to the output images of the PSR algorithm. Figures 22(d) through 22(f) show that PSR-UAV outperforms all other algorithms on average. The IMR of PSR-UAV is maximum at 83% in Figure 22(d), 77% in Figure 22(e), and 73% in Figure 22(f) for the minimum target density of 3.2 targets/\(\text{m}^2\). The IMR decreases with target density to a minimum of 67% in Figure 22(d), 31% in Figure 22(e), and 44% in Figure 22(f). The next algorithm to perform best is PSR-ICP. Other algorithms even fail to match any images correctly. Their IMR is 0%, which is indicated by the absence of color-coded bars in Figures 22(d) through 22(f). For example, ICDO, FilterReg, PSR-probability, PSR-NDT, PSR-UAV-affine, and PSR-affine have an IMR of 0% for the first target distribution and maximum target density (i.e., Figure 22(d) at 6.4 targets/m2).

5.3 Distance between Targets Matched Correctly

Consider Figure 17(a), in which all targets that are covered by both images are matched correctly. Matched targets are indicated with the solid black lines that join them. The length of these lines correspond to the distance reported in Figures 22(g) through 22(i), which show the average distance between targets that are matched correctly over the nine target distributions considered in the experimental evaluation. It needs to be noted that the average distance is not reported in those cases in which the IMR is 0%. For example, as already discussed in Section 5.2, ICDO, FilterReg, PSR-probability, PSR-NDT, PSR-UAV-affine, and PSR-affine have an IMR of 0% for the first target distribution and maximum target density (i.e., Figure 22(d) at 6.4 targets/m2), so the average distance for these algorithms is not reported in Figure 22(g) at 6.4 targets/m2.
The first observation is that all averages are below 0.15 m, which is due to the fact that targets cannot be matched if the distance between them is larger than \(T_\text{th}\) in Equation (13).
The second observation is that PSR-UAV, PSR-UAV-rigid, and PSR-rigid have small average distance when compared with the other algorithms. This result implies that these algorithms modify the location of targets of the second image to be almost the same location of the targets of the first image that they are matched with. This is possible due to the rigid transformation of the second image that these algorithms perform. This result is a consequence of the UAVs’ errors in traveled distance, altitude, pitch, yaw, and roll. The errors tend to be larger on traveled distance, altitude, and yaw, and these errors induce rigid transformations only. Errors in pitch and roll are small because all pictures are taken while the UAVs hover in the air. The average distance for PSR-UAV also remains constant as the target density increases. This is an indication that when targets are matched correctly, the scaling process described in Section 4.3.4 achieves a similar level of performance regardless of target density.
The last observation is that the average distance of PSR-UAV is less than the average distance of GPS-only, which implies that, first, errors in altitude, pitch, and roll of the UAVs affect the distance between targets matched correctly, and second, the scaling process is able to decrease this effect, but it does not eliminate it completely. For example, the image of a square on the ground turns into an isosceles trapezoid when the pitch or roll of the UAV is not zero, and scaling does not transform the trapezoid back into the original square.

6 Discussion of Results

The reason that the average TMR and IMR decrease with target density for PSR-UAV is that the probability of identifying an incorrect target pattern increases with target density. There is a higher probability of having similar target patterns on the ground within the overlapping area of adjacent images when the target density is high. On the other hand, for all other algorithms, the average TMR varies less when the target density increases. This is an indication that the other algorithms are unable to match target pairs effectively. Their non-zero TMR is caused by targets that show on one image only. The TMR would be closer to zero if those targets were absent. For example, consider Figure 19, which shows a selected case of superior target matching for PSR-probabilistic. Figure 19(c) shows the targets perfectly matched. There are 10 targets total, which include 4 targets paired and 6 individual targets. GPS-only is able to keep the 6 individual targets correctly as shown in Figure 19(a), and it fails to pair the other 4 targets correctly, so 6 out of 10 targets are registered correctly as explained in Section 5.1. These 6 targets correspond to targets covered by one image only. These targets were not paired to any other target, which is correct. The TMR in this case is \(\frac{6}{10} \times 100 = 60\%\). This TMR value is greater than zero due to targets that showed in one image only; it is not due to targets matched correctly. This is the reason the TMR is never negligible for low-performing PSR algorithms in Figures 22(a) through 22(c), and it is also the reason that the average TMR varies less when the target density increases for the PSR algorithms that were used to compare our PSR-UAV algorithm. The percentage of targets that are covered by one image only from the total number of targets in the overlapping area is independent of the target density because targets are randomly distributed on the ground regardless of location within the search area.
Fig. 19.
Fig. 19. Selected case of superior target matching rate for the PSR-probabilistic algorithm.
According to Figure 22, the average TMR and IMR of PSR-UAV is always higher than those of GPS-only. On the other hand, the opposite is true for all other algorithms, which in several cases worsen the TMR and IMR on average. For example, in Figure 22(b), all algorithms reduce the TMR and IMR reached by GPS-only with the exception of PSR-UAV and PSR-ICP. PSR-ICP slightly outperforms GPS-only at target densities 3.2 and 4.8 targets/m2. The reason is that the algorithms perform well when the error in distance and direction of the UAV from one image to the next is small relative to the distance between adjacent targets on the ground. For example, Figure 18 to Figure 20 are examples that show cases when PSR-ICP, PSR-probabilistic, and PSR-rigid achieve a TMR that is larger than the TMR achieved by PSR-UAV. In these cases, the UAVs happen to locate the second image relative the first image with small error in distance and direction as shown in Figure 18(a), Figure 19(a), and Figure 20(a). Therefore, if the UAVs were capable of determining location and direction of images with accuracy high enough that the deviation of targets on overlapping images was smaller than the distance between adjacent targets on the ground, our PSR-UAV algorithm, in comparison to PSR-ICP, PSR-probabilistic, and PSR-rigid, would not perform as well as reported in Figure 22.
Fig. 20.
Fig. 20. Selected case of superior target matching rate for the PSR-rigid algorithm.
Fig. 21.
Fig. 21. Selected case of superior target matching rate for the PSR-affine algorithm.
Fig. 22.
Fig. 22. Comparison of the TMR of the PSR-UAV algorithm against other PSR algorithms.
PSR-ICP [8] also performs better than our PSR-UAV in some cases. Figure 18 shows one of them. The reason for the reduced performance of PSR-UAV is that it is vulnerable to finding target patterns (i.e., triangles, line segments, or points) of deviation less than the threshold before finding the correct target pattern, and the probability of this event increases with target density as previously discussed. The reason for this vulnerability is that PSR-UAV compares target patterns that are closer to the adjacent image first, and the first pattern of deviation less than the threshold is selected for calculating the spatial transformation that aligns the images (Section 4.3.1).
Another advantage of achieving higher TMR and IMR not mentioned so far is on locating targets on the ground. Targets that show in multiple images have GPS coordinates in each image. Therefore, a target has as many coordinates as there are images that cover the target. For example, consider the targets at the intersection of the second, third, sixth, and seventh images in Figure 7(c). These targets appear in these four images, so each target has four different GPS coordinates, one per image. Ideally, the coordinates would all be the same, but they are not, which is due to the errors in distance and direction of the UAV. If these targets are registered correctly, the four coordinates can be identified by the PSR algorithm, and the location of the targets can be estimated with higher accuracy. Figure 23 shows the target locations estimated by PSR-UAV along with the actual locations of the targets for the third group of target distributions (Figure 7(c)) at target density of 3.2 targets/m2. Estimated locations are denoted by x and actual locations are denoted by concentric circles. The estimated number and location of targets are in error due to the non-ideal IMR and the error in distance and direction of the UAV. For example, Figure 23 shows that the fifth, seventh, and eighth images are not matched correctly. The images are shown in dashed lines. All other images are matched correctly. Therefore, the estimated number and location of targets covered by the fifth, seventh, and eighth images deviate from the actual values to a greater extent when compared against targets covered only by images matched correctly (e.g., targets covered by the second and sixth images). However, even in the case of images matched correctly, there is error in the estimated location of targets, which is due to the error of GPS coordinates of the UAV.
Fig. 23.
Fig. 23. Location of targets estimated by the PSR-UAV algorithm vs. actual location of targets on the ground.
The reason images do not align perfectly is UAVs’ error in traveled distance, altitude, pitch, yaw, and roll. The distribution of the error in traveled distance was presented in Figure 5, and the distribution of the error in yaw corresponds to the angle deviations presented in Figure 13(b) and Figure 16(b). These errors were accounted for directly in the PSR-UAV algorithm via the centroid and angle deviations defined in Section 4.3. On the other hand, the errors in altitude, pitch, and roll are all accounted for indirectly via the length deviation also defined in Section 4.3. The length deviation corresponds to the difference in length of line segments that join adjacent targets on the ground as seen from two different images. These line segments have the same length in all images if the pitch, roll, and altitude of the UAV do not change across images, making the length deviation zero. The mean and standard deviation of the centroid, angle, and length deviations are presented in Table 1. The centroid and angle deviations are not negligible, which can be clearly seen in Figure 7. On the other hand, the length deviation’s mean and standard deviation do not exceed 4 cm, which is a length that is negligible relative to the distance between adjacent targets on the ground. Therefore, the error in altitude, pitch, and roll was not significant. This is a consequence of capturing images while hovering in the air. The pitch and roll have significant variations when UAVs experience acceleration, so the PSR-UAV algorithm needs further development for application scenarios when images are captured from moving UAVs.

7 Conclusion

Two UAVs flew autonomously along a preprogrammed flight path making periodic stops along the way to capture images of targets on the ground that are randomly distributed. The literature review on the use of UAVs for the detection of ground objects such as cars, plants, or people demonstrated that it is reasonable to assume the random target distribution considered in this article. The UAVs’ flight path was such that adjacent images partially overlapped to avoid sections of the area of interest on the ground not being covered by any image. The wind on the UAVs and limited accuracy of UAVs’ instrumentation distort the alignment of images. The PSR-UAV algorithm was developed to realign the images by performing point set registration of targets detected on the images. The algorithm identifies patterns of up to three targets that are sufficiently similar to assume the patterns belong to the same set of targets on the ground. After identifying a matching set of targets, PSR-UAV determines an image transformation to match the identified set of targets on the overlapping area of the images. The performance of PSR-UAV was compared against six well-known algorithms: Iterative Closest Point, Coherent Point Drift, Probabilistic Data Association, Normal Distribution Transform, FilterReg, and Inverse Composition Discriminative Optimization. The results indicated superior performance from the PSR-UAV algorithm on average for the type of images and random target distributions considered in this article.
Future work includes the extension of the PSR-UAV algorithm to identify patterns of more than three targets on overlapping images. The effect of more complex flight paths (e.g., zigzag) that avoid physical obstacles such as trees also needs to be addressed. Finally, the effect of false positives and negatives of the target detection process on the PSR-UAV algorithm’s performance needs to be assessed.

References

[1]
3DR Robotics, 3DR Solo Drone. (n.d.). https://3dr.com/solo-drone/.
[3]
Raspberry Pi Foundation. (n.d.). Raspberry Pi Camera Module v2. (n.d.). https://www.raspberrypi.org/products/camera-module-v2/.
[4]
G. Agamennoni, S. Fontana, R. Y. Siegwart, and D. G. Sorrenti. (n.d.). Point Clouds Registration with Probabilistic Data Association. https://github.com/iralabdisco/probabilistic_point_clouds_registration.
[5]
G. Agamennoni, S. Fontana, R. Y. Siegwart, and D. G. Sorrenti. 2016. Point clouds registration with probabilistic data association. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’16). 4092–4098.
[6]
Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. 2006. SURF: Speeded up robust features. In Computer Vision (ECCV’06), Aleš Leonardis, Horst Bischof, and Axel Pinz (Eds.). Springer, Berlin,404–417.
[7]
Y. Bazi, S. Malek, N. Alajlan, and H. AlHichri. 2014. An automatic approach for palm tree counting in UAV images. In 2014 IEEE Geoscience and Remote Sensing Symposium. 537–540. DOI:
[8]
P. J. Besl and N. D. McKay. 1992. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 2 (Feb.1992), 239–256. DOI:
[9]
C. Bodensteiner, W. Hübner, K. Jüngling, P. Solbrig, and M. Arens. 2011. Monocular camera trajectory optimization using liDAR data. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops’11). 2018–2025. DOI:
[10]
A. B. Chan and N. Vasconcelos. 2012. Counting people with low-level features and Bayesian regression. IEEE Transactions on Image Processing 21, 4 (April2012), 2160–2177. DOI:
[11]
J. Chen, L. Luo, S. Wang, and H. Wu. 2018. Automatic panoramic UAV image mosaic using ORB features and robust transformation estimation. In 2018 37th Chinese Control Conference (CCC’18). 4265–4270. DOI:
[12]
Y. Chen, Z. Lin, X. Zhao, G. Wang, and Y. Gu. 2014. Deep learning-based classification of hyperspectral data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 7, 6 (June2014), 2094–2107. DOI:
[13]
D. Darji and G. Vejarano. 2018. Counting static targets using an unmanned aerial vehicle on-the-fly and autonomously. In 2018 15th Conference on Computer and Robot Vision (CRV’18). 206–213. DOI:
[14]
S. Du, N. Zheng, S. Ying, and J. Wei. 2007. ICP with bounded scale for registration of m-d point sets. In 2007 IEEE International Conference on Multimedia and Expo (ICME’07). 1291–1294. DOI:
[15]
Martin A. Fischler and Robert C. Bolles. 1981. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6 (June1981), 381–395. DOI:
[16]
Wei Gao and Russ Tedrake. 2019. FilterReg: Robust and efficient probabilistic point-set registration using Gaussian filter and twist parameterization. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19). 11087–11096. DOI:
[17]
W. Ge and R. T. Collins. 2009. Marked point processes for crowd counting. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’09). 2913–2920. DOI:
[18]
W. Han, D. Huang, and C. Xu. 2017. A novel compensation algorithm of aerial image registration. In 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP’17). 223–227. DOI:
[19]
J. Hermans, D. Smeets, D. Vandermeulen, and P. Suetens. 2011. Robust point set registration using EM-ICP with information-theoretically optimal outlier handling. In 2011 Conference on Computer Vision and Pattern Recognition (CVPR’11). 2465–2472. DOI:
[20]
S. Hinz. 2003. Detection and counting of cars in aerial images. In Proceedings 2003 International Conference on Image Processing (ICIP’03). III–997. DOI:
[21]
M. Israel and A. Reinhard. 2017. Detecting nests of lapwing birds with the aid of a small unmanned aerial vehicle with thermal camera. In 2017 International Conference on Unmanned Aircraft Systems (ICUAS’17). 1199–1207. DOI:
[22]
B. Jian and B. C. Vemuri. 2011. Robust point set registration using gaussian mixture models. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 8 (Aug.2011), 1633–1645. DOI:
[23]
R. Jonker and A. Volgenant. 1987. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing 38, 4 (Dec.1987), 325–340. DOI:
[24]
David G. Lowe. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 2 (Nov.2004), 91–110. DOI:
[25]
M. Lu, J. Zhao, Y. Guo, and Y. Ma. 2016. Accelerated coherent point drift for automatic three-dimensional point cloud registration. IEEE Geoscience and Remote Sensing Letters 13, 2 (Feb.2016), 162–166. DOI:
[26]
Jiayi Ma, Ji Zhao, Junjun Jiang, Huabing Zhou, and Xiaojie Guo. 2019. Locality preserving matching. International Journal of Computer Vision 127, 5 (May2019), 512–531. DOI:
[27]
[28]
T. Moranduzzo and F. Melgani. 2014. Automatic car counting method for unmanned aerial vehicle images. IEEE Transactions on Geoscience and Remote Sensing 52, 3 (March2014), 1635–1647. DOI:
[29]
A. Myronenko and X. Song. 2010. Point set registration: Coherent point drift. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 12 (Dec.2010), 2262–2275. DOI:
[30]
A. Pan and Y. Yang. 2016. Remote sensing images registration with different viewpoints. In 2016 International Conference on Audio, Language and Image Processing (ICALIP’16). 699–704. DOI:
[31]
C. C. C. Pang, W. W. L. Lam, and N. H. C. Yung. 2007. A method for vehicle count in the presence of multiple-vehicle occlusions in traffic images. IEEE Transactions on Intelligent Transportation Systems 8, 3 (Sept.2007), 441–459. DOI:
[32]
M. D. Phung, C. H. Quach, D. T. Chu, N. Q. Nguyen, T. H. Dinh, and Q. P. Ha. 2016. Automatic interpretation of unordered point cloud data for UAV navigation in construction. In 2016 14th International Conference on Control, Automation, Robotics and Vision (ICARCV’16). 1–6. DOI:
[33]
M. D. Pritt. 2014. Fast orthorectified mosaics of thousands of aerial photographs from small UAVs. In 2014 IEEE Applied Imagery Pattern Recognition Workshop (AIPR’14). 1–8. DOI:
[34]
D. Ramanan, D. A. Forsyth, and A. Zisserman. 2007. Tracking people by learning their appearance. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 1 (Jan.2007), 65–81. DOI:
[35]
Ignacio Rocco, Mircea Cimpoi, Relja Arandjelovic, Akihiko Torii, Tomás Pajdla, and Josef Sivic. 2018. Neighbourhood consensus networks. CoRR abs/1810.10510 (2018). arXiv:1810.10510. http://arxiv.org/abs/1810.10510.
[36]
E. Rosten and T. Drummond. 2005. Fusing points and lines for high performance tracking. In 10th IEEE International Conference on Computer Vision (ICCV’05) Volume 1, Vol. 2, 1508–1515. DOI:
[37]
E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. 2011. ORB: An efficient alternative to SIFT or SURF. In 2011 International Conference on Computer Vision (ICCV’11). 2564–2571. DOI:
[38]
D. Ryan, S. Denman, C. Fookes, and S. Sridharan. 2009. Crowd counting using multiple local features. In 2009 Digital Image Computing: Techniques and Applications (DICTA’09). 81–88. DOI:
[39]
T. Stoyanov, M. Magnusson, and A. J. Lilienthal. 2012. Point set registration through minimization of the L2 distance between 3D-NDT models. In 2012 IEEE International Conference on Robotics and Automation (ICRA’12). 5196–5201. DOI:
[40]
H. Tayara, K. G. Soo, and K. T. Chong. 2017. Vehicle detection and counting in high-resolution aerial images using convolutional regression neural network. IEEE Access PP, 99 (2017), 1–1. DOI:
[41]
E. Castelão Tetila, B. Brandoli Machado, N. A. d. S. Belete, D. A. Guimarães, and H. Pistori. 2017. Identification of soybean foliar diseases using unmanned aerial vehicle images. IEEE Geoscience and Remote Sensing Letters 14, 12 (Dec.2017), 2190–2194. DOI:
[42]
Jayakorn Vongkulbhisal, Beñat Irastorza Ugalde, Fernando De la Torre, and João P. Costeira. 2018. Inverse composition discriminative optimization for point cloud registration. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’18). 2993–3001. DOI:
[43]
J. A. Williams and M. Bennamoun. 2000. Evaluation of a novel multiple point set registration algorithm. In Proceedings 15th International Conference on Pattern Recognition (ICPR’00), Vol. 1, 1007–1010. DOI:
[44]
G. Xu, S. Du, and J. Xue. 2016. Precise 2D point set registration using iterative closest algorithm and correntropy. In 2016 International Joint Conference on Neural Networks (IJCNN’16). 4627–4631. DOI:
[45]
J. Yang, Q. Huang, B. Wu, and J. Chen. 2012. A remote sensing imagery automatic feature registration method based on mean-shift. In 2012 IEEE International Geoscience and Remote Sensing Symposium (IGARSS’12). 2364–2367. DOI:
[46]
Yaping Wang, Peihua Shao, and Yijin Chen. 2013. Comparison and analysis of transformation models on UAV image mosaic. In Proceedings 2013 International Conference on Mechatronic Sciences, Electric Engineering and Computer (MEC’13). 1042–1045. DOI:
[47]
R. Yu, Y. Yang, and K. Yang. 2018. Small UAV based multi-viewpoint image registration for extracting the information of cultivated land in the hills and mountains. In 2018 26th International Conference on Geoinformatics (CPGIS’18). 1–5. DOI:
[48]
C. Zeng and H. Ma. 2010. Robust head-shoulder detection by PCA-based multilevel HOG-LBP detector for people counting. In 2010 20th International Conference on Pattern Recognition (ICPR’10). 2069–2072. DOI:
[49]
Cong Zhang, Hongsheng Li, X. Wang, and Xiaokang Yang. 2015. Cross-scene crowd counting via deep convolutional neural networks. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 833–841. DOI:
[50]
H. Zhang, W. Ni, W. Yan, J. Wu, and S. Li. 2015. Robust SAR image registration based on edge matching and refined coherent point drift. IEEE Geoscience and Remote Sensing Letters 12, 10 (Oct.2015), 2115–2119. DOI:
[51]
Q. Zou, L. Ni, T. Zhang, and Q. Wang. 2015. Deep learning based feature selection for remote sensing scene classification. IEEE Geoscience and Remote Sensing Letters 12, 11 (Nov.2015), 2321–2325. DOI:

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Spatial Algorithms and Systems
ACM Transactions on Spatial Algorithms and Systems  Volume 9, Issue 3
September 2023
207 pages
ISSN:2374-0353
EISSN:2374-0361
DOI:10.1145/3597152
Issue’s Table of Contents
This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2023
Online AM: 04 March 2023
Accepted: 24 February 2023
Revised: 27 February 2022
Received: 16 November 2020
Published in TSAS Volume 9, Issue 3

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Point set registration
  2. unmanned aerial vehicle
  3. target tracking
  4. target localization

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 928
    Total Downloads
  • Downloads (Last 12 months)409
  • Downloads (Last 6 weeks)64
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Full Access

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media