The next step is to identify equivalent target patterns on the first and second images. Three patterns are considered: triangles, line segments, and points. First, the PSR-UAV algorithm attempts to find matching triangles. If this test fails, the algorithm attempts to find matching line segments, and if this test fails, the algorithm attempts to find matching points. If matching points are not found, the algorithm determines that the overlapping images do not have any common targets. On the other hand, matching patterns are an indication of common targets in the overlapping region of the images.
4.3.1 Matching Triangles.
A set of triangles is found for each image as shown in Figure
12. Let
\(N_1\) and
\(N_2\) be the number of overlapping targets of the first and second images, respectively. These targets are shown in white in Figure
12. Let
\(\lbrace \bigtriangleup _{i,1},\bigtriangleup _{i,2},....,\bigtriangleup _{i,N_i-2}\rbrace\) be the set of triangles of the
\(i^\text{th}\) image, where
\(i = 1,2\).
\(\bigtriangleup _{i,1}\) includes targets
\(\mathbf {p}_{i,1}\),
\(\mathbf {p}_{i,2}\), and
\(\mathbf {p}_{i,3}\);
\(\bigtriangleup _{i,2}\) includes targets
\(\mathbf {p}_{i,2}\),
\(\mathbf {p}_{i,3}\), and
\(\mathbf {p}_{i,4}\); and so on. The last triangle,
\(\bigtriangleup _{i,N_i-2}\), includes targets
\(\mathbf {p}_{i,N_i-2}\),
\(\mathbf {p}_{i,N_i-1}\), and
\(\mathbf {p}_{i,N_i}\). Two triangles are said to
match if they are sufficiently similar in the following manner. Let the
deviation between two triangles, denoted by
\(\bigtriangleup _\text{dev}\), be a metric of their difference. Two triangles match if their deviation
\(\bigtriangleup _\text{dev}\) is below a threshold measured experimentally. Let
\(\bigtriangleup _\text{dev}\) be defined as follows. Consider any two triangles
\(\bigtriangleup _{1,n_1}\) and
\(\bigtriangleup _{2,n_2}\), where
\(n_i = 1,2,\ldots ,N_i-2\), such as the two yellow triangles in Figure
12. Let
\(\lbrace l_{i,n_i,1},l_{i,n_i,2},l_{i,n_i,3}\rbrace\) be the length of the edges, from shortest to largest, of
\(\bigtriangleup _{i,n_i}\); let
\(\mathbf {b}_{i,n_i}\) be the angle bisector of the smallest angle of
\(\bigtriangleup _{i,n_i}\); and let
\(\mathbf {\Delta }_{i,n_i,j}\) be the coordinate of the
\(j^\text{th}\) vertex of
\(\bigtriangleup _{i,n_i}\). For example, the vertices of the first triangle of the
\(i^\text{th}\) image (i.e.,
\(\bigtriangleup _{i,1}\)) are
\(\mathbf {p}_{i,1}\),
\(\mathbf {p}_{i,2}\), and
\(\mathbf {p}_{i,3}\), so
\(\mathbf {\Delta }_{i,1,1} = \mathbf {p}_{i,1}\),
\(\mathbf {\Delta }_{i,1,2} = \mathbf {p}_{i,2}\), and
\(\mathbf {\Delta }_{i,1,3} = \mathbf {p}_{i,3}\). The deviation between triangles
\(\bigtriangleup _{1,n_1}\) and
\(\bigtriangleup _{2,n_2}\) is given in Equation (
2) as a function of the
centroid deviation,
length deviation, and
angle deviation between the triangles. The centroid deviation, denoted by
\(c_\text{dev}\), is the Euclidean distance between the centroids of
\(\bigtriangleup _{1,n_1}\) and
\(\bigtriangleup _{2,n_2}\) as defined in Equation (
3), where
\(\frac{1}{3}\sum _{j=1}^{3}\mathbf {\Delta }_{i,n_i,j}\) is the centroid of
\(\bigtriangleup _{i,n_i}\). The length deviation, denoted by
\(l_\text{dev}\), is the
root mean square (RMS) error between the lengths of the triangles’ edges, from shortest to largest, as defined in Equation (
4). The angle deviation, denoted by
\(\alpha _\text{dev}\), is the angle between the bisectors of the smallest angles of the triangles, and it is defined in Equation (
5), where
\(\mathbf {b}_{i,n_i}\) is the bisector of the smallest angle of
\(\bigtriangleup _{i,n_i}\). Figure
12 shows the bisectors of two matching triangles that are indicated in yellow.
Deviations
\(c_\text{dev}\),
\(l_\text{dev}\),
\(\alpha _\text{dev}\), and
\(\bigtriangleup _\text{dev}\) cannot be negative by definition, and they are all zero for perfectly located and aligned images, which is not true in practice. Therefore, the deviation metrics take on positive and random values whose distributions are reported in Figure
13. The distributions were obtained experimentally as follows. Consider three targets that show on two images such as those of the yellow triangles in Figure
12. Now, consider the deviation between the two triangles. This deviation is calculated for every two same-target triangles for every two images collected for the experimental evaluation described in Section
5. Figure
13(a) and Figure
13(b) show the distribution of
\(l_\text{dev}\) and
\(\alpha _\text{dev}\). The distribution of
\(c_\text{dev}\) is approximated to the distribution of the error in displacement shown in Figure
5 (i.e., the error in UAV’s direction is disregarded). The mean and standard deviation of these distributions are listed in Table
1, and their values are used to normalize each deviation metric so that
\(\bigtriangleup _\text{dev}\) can be defined as in Equation (
2), which is the RMS value of the normalized deviation metrics. Figure
13(c) shows the distribution of
\(\bigtriangleup _\text{dev}\).
Finally, triangles
\(\bigtriangleup _{1,n_1}\) and
\(\bigtriangleup _{2,n_2}\) match if their deviation
\(\bigtriangleup _\text{dev}\) is below the threshold defined in Equation (
6), which depends on the mean and standard deviation of the distribution of
\(\bigtriangleup _\text{dev}\) (Figure
13(c)) and constant
k, which was determined experimentally (
\(k = 1.55\)).
The calculation of
\(\alpha _\text{dev}\) requires an extra step not mentioned previously for clarity. The reason is that
\(\alpha _\text{dev}\) depends on identifying the minimum angle of triangles
\(\bigtriangleup _{1,n_1}\) and
\(\bigtriangleup _{2,n_2}\), and the minimum angle can be misidentified due to tilting of the UAV when capturing an image while hovering in the presence of wind. Consider any three targets covered in both images as shown in Figure
14.
\(\bigtriangleup _{1,n_1}\) is shown in green, and
\(\bigtriangleup _{2,n_2}\) is shown in blue. Assume that
\(\bigtriangleup _{1,n_1}\) and
\(\bigtriangleup _{2,n_2}\) correspond to the same targets on the ground and that they do not align in Figure
14(a) due to the error in direction and displacement of the UAV. Therefore,
\(\bigtriangleup _{1,n_1}\) and
\(\bigtriangleup _{2,n_2}\) are not on top of one another in Figure
14(a). The angles of
\(\bigtriangleup _{1,n_1}\) and
\(\bigtriangleup _{2,n_2}\) in Figures
14 and
15 are named after their relative values, so
\(\alpha _\text{s}\),
\(\alpha _\text{m}\), and
\(\alpha _\text{l}\) are the smallest, medium, and largest angles respectively. The smallest angle of
\(\bigtriangleup _{1,n_1}\) and
\(\bigtriangleup _{2,n_2}\) is identified correctly in Figure
14, and it is not in Figure
15. The smallest angle of
\(\bigtriangleup _{2,n_2}\) is mistaken for the medium angle in Figure
15(a), which can be observed by comparing
\(\alpha _\text{s}\) and
\(\alpha _\text{m}\) of color blue in Figure
14(a) and Figure
15(a). The tilted UAV causes a change of perspective in the image that causes the angles to change. In this case,
\(\alpha _\text{s}\) of
\(\bigtriangleup _{2,n_2}\) increases its value, and
\(\alpha _\text{m}\) decreases its value, becoming the smallest angle of
\(\bigtriangleup _{2,n_2}\).
The following test is performed to detect misidentification of the smallest angle.
\(\bigtriangleup _{2,n_2}\) is moved toward
\(\bigtriangleup _{1,n_1}\) to match the vertex of the largest angle as shown in Figure
14(b) and Figure
15(b).
\(\bigtriangleup _{2,n_2}\) is then rotated along this vertex until the edge that joins the vertices of
\(\alpha _\text{s}\) and
\(\alpha _\text{l}\) are colinear. This rotation is shown in Figure
14(c) and Figure
15(c). There can be two outcomes: (1)
\(\bigtriangleup _{1,n_1}\) and
\(\bigtriangleup _{2,n_2}\) overlap as shown in Figure
14(c); (2)
\(\bigtriangleup _{1,n_1}\) and
\(\bigtriangleup _{2,n_2}\) do not overlap as shown in Figure
15(c). Overlapping triangles as in Figure
14(c) is indication that
\(\alpha _\text{s}\) is identified correctly in both
\(\bigtriangleup _{1,n_1}\) and
\(\bigtriangleup _{2,n_2}\). Non-overlapping triangles as in Figure
15(c) is indication that
\(\alpha _\text{s}\) was misidentified in one of the triangles. The solution is to swap the vertices of
\(\alpha _\text{s}\) and
\(\alpha _\text{m}\) in
\(\bigtriangleup _{2,n_2}\), so the vertex of
\(\alpha _\text{m}\) is considered the vertex of the smallest angle, and the vertex of
\(\alpha _\text{s}\) is considered the vertex of the medium angle. Then, the same rotation is performed again, and if the triangles do not overlap once more, the vertices are reordered in the original sequence from the smallest angle to largest angle, and the vertices of
\(\alpha _\text{m}\) and
\(\alpha _\text{l}\) are swapped. The rotation is performed one last time, and if the triangles still do not overlap, the vertices are reordered in the original sequence from the smallest angle to largest angle again, and the vertices of
\(\alpha _\text{s}\) and
\(\alpha _\text{l}\) are swapped. The bisector is determined on the first vertex of the reordered list of angles.
The deviation among triangles is calculated in the following order:
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,1},\bigtriangleup _{2,1})\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,1},\) \(\bigtriangleup _{2,2})\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,2},\bigtriangleup _{2,1})\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,1},\bigtriangleup _{2,3})\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,3},\bigtriangleup _{2,1})\),
\(\ldots\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,2},\bigtriangleup _{2,2})\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,2},\bigtriangleup _{2,3})\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,3},\bigtriangleup _{2,2})\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,2},\bigtriangleup _{2,4})\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,4},\bigtriangleup _{2,2})\),
\(\ldots\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,N_1-3},\bigtriangleup _{2,N_2-3})\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,N_1-3},\bigtriangleup _{2,N_2-2})\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,N_1-2},\bigtriangleup _{2,N_2-3})\),
\(\bigtriangleup _\text{dev}(\bigtriangleup _{1,N_1-2},\bigtriangleup _{2,N_2-2})\), and the first deviation to be found below the threshold
\(\bigtriangleup _\text{th}\) is used to perform point set registration on the two images as will be explained in Section
4.3.4. The reason for this order is that the first triangle of an image is the closest triangle to the center of the other image, and the closer it is, the higher the probability that the triangle is covered by both images. Therefore, the first comparison is between the first triangle of the first image and the first triangle of the second image (i.e.,
\(\bigtriangleup _{1,1}\) and
\(\bigtriangleup _{2,1}\)). The second comparison is between the first triangle of the first image and the second triangle of the second image (i.e.,
\(\bigtriangleup _{1,1}\) and
\(\bigtriangleup _{2,2}\)). In the third comparison, the images alternate, so the comparison is between the first triangle of the second image and the second triangle of the first image (i.e.,
\(\bigtriangleup _{1,2}\) and
\(\bigtriangleup _{2,1}\)). The images alternate again in the fourth comparison, so the comparison is between the first triangle of the first image and the third triangle of the second image (i.e.,
\(\bigtriangleup _{1,1}\) and
\(\bigtriangleup _{2,3}\)). The images alternate once again in the fifth comparison, so the comparison is between the first triangle of the second image and the third triangle of the first image (i.e,
\(\bigtriangleup _{1,3}\) and
\(\bigtriangleup _{2,1}\)), and so on.
The PSR-UAV algorithm switches to finding matching segments if all deviations among triangles are above \(\bigtriangleup _\text{th}\).
4.3.2 Matching Segments.
Let
\(\lbrace S_{i,1}, S_{i,2}, \dots , S_{i,N_i-1}\rbrace\) be the set of segments of the
\(i^\text{th}\) image, where
\(i = 1, 2\).
\(S_{i,1}\) is the line segment that joins targets
\(\mathbf {p}_{i,1}\) and
\(\mathbf {p}_{i,2}\);
\(S_{i,2}\) is the line segment that joins targets
\(\mathbf {p}_{i,2}\) and
\(\mathbf {p}_{i,3}\); and so on. The last segment,
\(S_{i,N_i-1}\), is the line segment that joins targets
\(\mathbf {p}_{i,N_i-1}\) and
\(\mathbf {p}_{i,N_i}\). The deviation between two segments, denoted by
\(S_\text{dev}\), is defined in the same manner as the deviation between triangles in Equation (
2). The centroid deviation for segments is defined in Equation (
7), where
\(\mathbf {s}_{i,n_i,j}\) is the coordinate of the
\(j^\text{th}\) end of
\(\mathbf {s}_{i,n_i}\). For example, the two ends of the first segment of the
\(i^\text{th}\) image (i.e.,
\(\mathbf {s}_{i,1}\)) are
\(\mathbf {s}_{i,1,1} = \mathbf {p}_{i,1}\) and
\(\mathbf {s}_{i,1,2} = \mathbf {p}_{i,2}\), and the two ends of the second segment of the
\(i^\text{th}\) image (i.e.,
\(\mathbf {s}_{i,2}\)) are
\(\mathbf {s}_{i,2,1} = \mathbf {p}_{i,2}\) and
\(\mathbf {s}_{i,2,2} = \mathbf {p}_{i,3}\), and so on. The length deviation is defined in Equation (
8) as the absolute value of the difference of segment lengths. The angle deviation is defined in Equation (
9). It is the smallest angle between the two lines that correspond to the segments.
The distributions of the deviation metrics are shown in Figure
16. They were found experimentally in the same way that they were found for triangles as described in Section
4.3.1. Figures
16(a) through
16(c) show the distribution of
\(l_\text{dev}\),
\(\alpha _\text{dev}\), and
\(S_\text{dev}\), respectively. The distribution of
\(c_\text{dev}\) is approximated to the distribution of the error in displacement, which is shown in Figure
5 (i.e., the error in UAV’s direction is disregarded). The mean and standard deviation of these distributions are listed in Table
1, and their values are used to normalize each deviation metric so that
\(S_\text{dev}\) can be defined in the same way of
\(\bigtriangleup _\text{dev}\) in Equation (
2).
\(S_{1,n_1}\) and
\(S_{2,n_2}\) match if their deviation,
\(S_\text{dev}(S_{1,n_1},S_{2,n_2})\), is below the threshold
\(S_\text{th}\) given in Equation (
10), which depends on the mean and standard deviation of the distribution of
\(S_\text{dev}\) shown in Figure
16(c).
The deviation among segments is calculated in the same order of triangles, and the first deviation to be found below the threshold
\(S_\text{th}\) is used to perform point set registration on the two images as will be explained in Section
4.3.4. The PSR-UAV algorithm switches to finding matching points if all deviations among segments are above
\(S_\text{th}\).
4.3.4 Point Set Registration.
If a matching target pattern (i.e., triangle, line segment, point) is identified, the next step in the PSR-UAV algorithm is to perform a rigid transformation (i.e., translation, rotation, and scaling) on the second image so that the correspondence between targets of the two images can be established.
The location of the second image is corrected by moving the image (i.e., translation) by the length and direction of the vector that starts at the centroid of the matching pattern on the second image and ends at the centroid of the matching pattern on the first image. For example, consider the image pair shown in Figure
12. The matching target pattern is the yellow triangles. Then, translation of the second image takes place by moving it toward the first image until the centroids of the yellow triangles match. The orientation and scale of the second image are also corrected if the matching patterns are triangles or line segments. The orientation is corrected by rotating the second image around the centroid of its matching pattern. The angle of rotation is the angle deviation of the matching patterns (i.e.,
\(\alpha _\text{dev}\)), and the direction of rotation is such that
\(\alpha _\text{dev}\) is zero after rotation. For example, the second image of Figure
12 is rotated until the angle bisectors of the yellow triangles (i.e., vectors
\({\bf b_{1,1}}\) and
\({\bf b_{2,11}}\)) are colinear. The image is then scaled so that the mean square error of the distance between corresponding targets of the matching patterns is minimized. In Figure
12, it is the mean square error of the distance between
\({\bf p_{1,2}}\) and
\({\bf p_{2,12}}\),
\({\bf p_{1,1}}\) and
\({\bf p_{2,13}}\), and
\({\bf p_{1,3}}\) and
\({\bf p_{2,11}}\) after the second image is translated and rotated.
The next step is to determine the correspondence between targets. Let a
target pair be defined as a set of two targets, one target from the first image and the other target from the second image. Let the distance of a target pair be the Euclidean distance between the targets of the pair. The target pair of shortest distance is found first. The next target pair of shortest distance is then found, omitting the pair previously found. This process continues until the distance is greater than the threshold
\(T_\text{th}\) defined in Equation (
13), where
\(\overline{l_\text{dev}}\) and
\(\sigma _{l_\text{dev}}\) are the mean and standard deviation of the length deviation of line segments. The distribution of this deviation is shown in Figure
16(a).
Finally, target pairs of distance less than \(T_\text{th}\) are considered to be one target only on the ground, and we refer to them as matched target pairs. Target pairs of distance greater than \(T_\text{th}\) are considered to be two different targets.