Real-time multi-resolution edge detection with pattern analysis on graphics processing unit

Abstract

Edge detection is defined as the process of detecting and representing the presences of and locations of image signal discontinuities, which serves as the basic transformation of signals into symbols and it influences the performance of subsequent higher level pattern analysis downstream. In general, the edge detection operation has two main steps: filtering, and detection and localization. In the first step, finding an optimal scale of the filter is an ill-posed problem, especially when a single—global—scale is used over the entire image. Multi-resolution description of the image which can represent the image features occurring in a range of scales is used, where a combination of Gaussian filters with different scales can ameliorate the single scale issue. In the second step, often edge detectors have been designed to capture simple ideal step functions in image data, but real image signal discontinuities deviate from this ideal form. Another three types of deviations from the step function which relate to real distortions occurring in natural images are examined. These types are impulse, ramp, and sigmoid functions which, respectively, represent narrow line signals, simplified blur effects, and more accurate blur modeling. General rules for edge representation based upon the classification of edge types into four categories—ramp, impulse, step, and sigmoid (RISS) are developed from this analysis. Additionally, the proposed algorithm performs connectivity analysis (CA) on edge map to ensure that small, disconnected edges are removed. The performance analysis on experiments supports that the proposed multi-resolution edge detection algorithm with edge pattern analysis does lead to more effective edge detection and localization with improved accuracies. To expand the proposed algorithm into real-time applications, a parallel implementation on a graphics processing unit (GPU) is presented in this paper. For the various configurations in our test, the GPU accelerated RISS shows a scalable speedup as the resolution of an image increases. We also achieved 20 frames per second in video processing ($640\times 480$).

Appendix: Edge pattern analysis

Edge patterns are classified according to their intensity profiles. In this appendix, the behaviors of basic edge patterns on the different order derivatives are examined. Deduction process will be shown for each pattern. Then, the final detection and localization rules for each edge pattern will display.

1.1 Impulse edge

In digital image processing, impulse edges are extrema in local area. If an impulse edge is isolated, it will be generally treated as an impulse noise generated during imaging process. Actually, lines correspond to local extreme of the gray level image and are of great use in the identification of image features, such as roads and runway, which are composed by impulse edges. For example, an ideal line with 1 pixel width is represented by a group of impulse edges, where the number of the impulse edge depends on the length of the ideal line. A general impulse edge is shown in Fig. 15. $x$ is the middle of the pulse. $U_{0} = f(x)-f(x-1)$ and $U_{1} = f(x)-f(x+1)$ are, respectively, the differences between $x$ and its two adjacent neighbors $x-1$ and $x+1$. $U_{0}$ and $U_{1}$ have same sign (positive or negative).

Fig. 15

An impulse edge

Then, lets examine the extremum rules for an impulse edge. Assume the intensity value of $x$ is larger than its neighbors. The gradient $f'(x)$ at the point $x$ is equivalent with $-U_{1}$, since $f'(x) = f(x+1)-f(x)$. The first rule for an extremum is $|U_{1}| \ge \text{Threshold}$. If the $\text{Threshold}$ is set properly, the first rule can be satisfied. After that, the second rule for an extremum needs to be examined, which is $f''(x)f''(x+1) < 0$. Remember

$$\begin{aligned}&f''(x)f''(x + 1) = [f'(x) - f'(x - 1)][f'(x + 1) - f'(x)]\nonumber \\&= [f'(x)- f'(x- 1)][f(x + 2) - f(x + 1) - f'(x)]\\&= [f'(x - 1) - f'(x)][f'(x) + f(x + 1) - f(x + 2)]\nonumber \\&= AB,\nonumber \end{aligned}$$

(9)

where $A = f'(x- 1)- f'(x)$ and $B = f'(x) + f(x + 1) - f(x + 2)$.

For an impulse edge, $A = U_{0} +U_{1}$ and $B = -(U_{1} + f'(x+1))$. The sign of $A$ is determined by sign of $U_{1}$, since $U_{0}$ and $U_{1}$ have same sign. If the impulse edge connects with flat area, it means $f(x + 2)$ is equivalent with $f(x + 1)$. Then, $f'(x+1) = f(x+2) - f(x+1)$ = 0. The sign of $B$ is determined by $U_{1}$ too. Therefore, $A$ and $B$ have opposite sign, which proved $f''(x)f''(x + 1) < 0$. Even for more general conditions, the second rule can satisfy. Assume $f'(x)$ is less than 0, which means that $f(x + 1) - f(x)$ is less than 0. Then, we can get $f(x + 2) \ge f(x + 1)$, otherwise, we get a continuous rising or decreasing profile, which is treated as a ramp or a sigmoid edge in this paper. For this reason, $f'(x + 1) = f(x+2)-f(x+1)$ has same sign as $U_{1}$, namely, $f'(x)$, which makes the signs of $A$ and $B$ still opposite.

For the third rule, the requirement is $f'(x)f'''(x) < 0$, where $f'(x) = -U_1$ and $f'''(x) = f'(x+1)-f'(x)-(f'(x) - f'(x-1))$. And,

$$\begin{aligned} f'''(x)&= f'(x + 1) + f'(x - 1) - 2f'(x)\\&= f'(x + 1) + U_{0} + 2U_{1}.\nonumber \end{aligned}$$

(10)

If the impulse edge is in a flat area, $f'(x + 1) = f(x + 2) - f(x+1)$ is zero. Therefore, $f'(x)f'''(x) = -U_{1}(U_{0}+2U_{1}) < 0$. For more general cases, the impulse edge still satisfies this condition, for continuous rising or decreasing profile is not allowed in the impulse edge pattern. This makes $f'(x + 1)$ has the same sign as $U_{0}$ and $U_{1}$.

Now, we theoretically proved that the impulse edge can be detected and localized by the extremum approach. However, the $x-1$ point in the impulse edge pattern can also satisfy all three rules, only if $|U_{0}| \ge \text{Threshold}$. This is the double edge response introduced by the second-order derivative, which also produces a thick edge for an impulse edge as the first-order derivative. This effect reduces the extremum approach detection and localization accuracy. For example, if the edge detector encounters a group of impulses, it will consider all impulses and their forward adjacent pixels as edges.

After analysis, an impulse edge detector is designed to examine the point $x$ as follows,

• 1. The extremum approach detector examines $x$ and its neighbors from all four orientations.

• 2. $x$ and its former pixel $x - 1$ are both extrema at least from two orientations.

• 3. The point $x$ is labeled as an edge.

In this paper, only the locations of impulses are considered as edges, which can guarantee a single response and reduce detection and localization error. The reason that the impulse edge should be detected from at least two orientation is to avoid the corner of a step edge is considered as an impulse.

1.2 Step edge

A step edge involves a transition between two intensity levels occurring ideally over the distance of one pixel. Step edges frequently occur in the computer generated images, whose clean and ideal prosperities are used to make them ideal edge models in algorithm development. A general step edge is shown in Fig. 16. In ideal design, the edge point should be labeled at the middle of the onset and the end of the step edge. Without interpolation process or magnified the original size of being processed image, the onset of the step edge is commonly chosen as an edge point.

Fig. 16

A step edge

In the step edge model, $x$ is the onset point. $U_{0} = f(x + 1) - f(x)$ can be positive or negative. The gradient of $x$ is equivalent with $|U_{0}|$. The first rule for extremum is $|U_{1}|\ge \text{Threshold}$. If the $\text{Threshold}$ is set properly, the first rule can be satisfied. Then, the sign of second derivative between the onset and the end points should be different:

$$\begin{aligned}&f''(x)f''(x + 1) = [f'(x) - f'(x - 1)][f'(x + 1) - f'(x)]\nonumber \\&= -[f'(x) - f'(x - 1)][f'(x) - f'(x + 1)]\\&= -AB,\nonumber \end{aligned}$$

(11)

where $A = f'(x)-f'(x-1)$ and $B = f'(x)-f'(x+1)$. For a step edge, $A = U_{0} - f'(x - 1)$ and $B = U_{0} - f'(x + 1)$.

For an ideal step edge, $f'(x - 1)$ and $f'(x + 1)$ should be zero, when $f(x - 1) = f(x)$ and $f(x + 1) = f(x + 2)$. The signs of $A$ and $B$ are both only determined by sign of $U_{0}$. Therefore, $f''(x)f''(x+1) < 0$ is proved. To prove the second rule for general cases, the second rule is also proved as similar procedure stated in the impulse edge pattern section. Assume $U_{0} > 0$. Under this condition, $f'(x - 1) = f(x) - f(x - 1)$ and $f'(x+1) = f(x+2)-f(x+1)$ should be less than zero, which guarantees $A$ and $B$ have same signs. Otherwise, the pixel x should exist at a ramp or sigmoid edge pattern. For the third rule, the requirement is $f'(x)f'''(x) < 0$, where $f'(x) = U_{0}$ and $f'''(x) = f'(x+1) - f'(x) - (f'(x) - f'(x - 1))$. The $f'''(x)$ can be represented as

$$\begin{aligned} f'''(x)&= f'(x + 1) + f'(x- 1) - 2f'(x)\\&= f'(x + 1) + f'(x - 1) - 2U_{0}.\nonumber \end{aligned}$$

(12)

For an ideal step edge, $f'(x + 1)$ and $f'(x - 1)$ are zero. Therefore, $f'(x)f'''(x) = -2U^{2}_{0} < 0$. For more general cases, since all continuous rising or decreasing profile are considered as ramp or sigmoid edges, $f'(x+1)$ and $f'(x-1)$ still do not change the sign of $f'''(x)$ determined by $-2U_{0}$. Therefore, the third rule still works for the step edge pattern.

After theoretically testified, the step edge satisfies local extremum conditions. To further differentiate it from other edge patterns, the step edge detector is designed to examine the point x as,

• 1. $x$ is a local extremum, and its former pixel $x - 1$ and behind pixel $x+1$ are not local extremum examined by the extremum rules.

• 2. The point $x$ is labeled as an edge.

1.3 Ramp edge

In practice, digital images have edges that are blurred and noisy, with the degree of blurring determined principally by limitations in the focusing mechanism, and the noise level determined principally by the electronic components of the imaging system. In such situations, edges are more closely modeled as having an intensity ramp profile. The ideal ramp profile is drawn in Fig. 17. The slope of the ramp is inversely proportional to the degree of blurring in the edge. In this paper, the span between any adjacent point in the ramp slope is set as constant. Otherwise, the edge will be considered as a sigmoid edge, whose pixels have different span in the transition slope. For a ramp edge, edges should be only the onset and the end of the ramp edge.

Fig. 17

A ramp edge

Also, the extremum rules are examined for the ramp edge pattern. $x$ is the onset point of the ramp, and $x + n + 1$ or $x'$ is the end of the ramp. $n$ is the number points in the profile of the ramp edge. $U_{0}$ is gotten by $f(x + n + 1) - f(x)$, and the ramp slope is $m = U_{0}/(n + 1)$. To examine the first rule of the extremum, the gradient of a ramp edge $|U_{0}|$ should not be less than a threshold, which is equivalent to $|U_{0}|\ge \text{Threshold}$. Different from the impulse edge and the step edge, the ramp edge does not ensure that adjacent opposite second-order derivative points exist. Since the span, or call it gradient, in the ramp slope transition is constant, $f''(x + i)$ always equals with 0, where $1 \le i \le n$. Thus, when the second rule of the extremum is applied into the ramp edge, it is adjusted as $f''(x)f''(x + n + 1) = f''(x)f''(x') < 0$.

$$\begin{aligned}&f''(x)f''(x') \nonumber \\&= [f'(x) - f'(x - 1)][f'(x') - f'(x' - 1)]\\&= [f'(x) - f'(x - 1)][f'(x + n) - f'(x + n - 1)]\nonumber \\&= -AB\nonumber \end{aligned}$$

(13)

where $A = f'(x)-f'(x-1)$ and $B = f'(x+n)-f'(x+n-1)$. In ideal condition, the ends of the ramp edge is flat area. Therefore, $f'(x-1)$ and $f'(x+n)$ should be zero. The signs of $A$ and $B$ are determined by signs of $f'(x)$ and $f'(x+n-1)$. Since $f'(x) = f(x+1)-f(x)$ and $f'(x+n-1) = f(x+n)-f(x + n - 1)$ have the same sign of $U_{0}$, $f''(x)f''(x + 1) < 0$ is proved. Then, extend this deduction to general cases, where the ends of the ramp is not flat segments. Therefore, $f'(x-1)$ and $f'(x + n)$ are not zero. Assume $U_{0}$ is larger than zero, therefore $f'(x - 1) = f(x) - f(x - 1)$ and $f'(x + n) = f(x + n + 1) - f(x + n)$ should be less than 0, which makes $A$ and $B$ have same sign again. Otherwise, the profile of the ramp edge is not only $n$ points, for the profile is still rising or decreasing beyond $x$ and $x'$ points.

For the third rule, the ramp edge requires

$$\begin{aligned}&f'(x)f'''(x) < 0,\\&f'(x')f'''(x') < 0.\nonumber \end{aligned}$$

(14)

$f'(x)$ has the same sign as $U_{0}$, and $f'''(x) = f'(x + 1) - f'(x)-(f'(x)-f'(x-1))$. For $f'''(x)$, it can be represented as

$$\begin{aligned} f'''(x)&= f'(x + 1) + f'(x - 1) - 2f'(x)\nonumber \\&= U_{0}/(n + 1) + f'(x - 1) - 2U_{0}/(n + 1)\\&= f'(x - 1) - U_{0}/(n + 1).\nonumber \end{aligned}$$

(15)

If $f(x - 1)$ equals with $f(x)$, $f'(x - 1)$ is zero. Therefore, $f'(x)f'''(x) = -2(U_{0}/(n + 1))^2 < 0$. For more general cases, $f'(x - 1)$ should still be negative, since $x$ is the onset of the profile. Then, the sign of $f'''(x)$ is still opposite from $f'(x)$, which makes the condition $f'(x)f'''(x) < 0$ satisfied.

But, for the end point of the ramp edge, the third rule of the extremum approach is not guaranteed. To testify $f'(x')f'''(x') < 0$, the sign of both $f'(x')$ and $f'''(x')$ are required. And,

$$\begin{aligned} f'(x') = f'(x + n + 1) = f(x + n + 2)- f(x + n + 1). \end{aligned}$$

(16)

Though in general cases, $f(x + n + 2)$ should be less than $f(x + n + 1)$, for $U_{0}$ is assumed positive. But, in ideal case, $f(x + n + 2)$ equal with $f(x + n + 1)$. Then, the first-order derivative of the end point $f'(x')$ will be zero. Furthermore, even the first derivative might be not zero in general cases, the third-order derivative $f'''(x') = f'(x'+1)+f'(x'-1)-2f'(x')$ is also not determinative by similar deduction. Since $f'(x')$ must have opposite sign as $U_{0}$, and $f'(x'-1)$ have same sign as $U_{0}$, the sign of $f'''(x')$ is determined by $f'(x' + 1)$. However, the $f'(x'+1)$ can be any value. Therefore, the end of the ramp edge cannot be guaranteed the third rule of finding extremum. Therefore, the third rule does not work for the ramp edge, which also pointed out that Clarks extremum method is not applicable to all edge patterns.

Different from other edge patterns, the ramp edge is a special one with some special characteristics. Therefore, its detector is designed as,

• 1. The first-order derivative of all points in the ramp edge must have same sign, except the end of the ramp edge.

• 2. The gradients of the points in the slope of the ramp edge should be constant.

• 3. The number of minimal points in the slope is 2.

• 4. The absolute value of the difference between the onset and the end points should be larger than the pre-defined threshold.

• 5. The onset and the end points are labeled as edges.

The reason for abandoning the second rule of extremum approach is that the ideal ramp can guarantee satisfying it. Therefore, the second rule is not added in the detector.

1.4 Sigmoid edge

In a real or natural image, the sigmoid edge is more common than others, due to focal blur, penumbral blur, object shape, etc., [10]. To complement the blurring effect and the design defect, the sigmoid edge is adopted as a basic edge pattern in this paper. The sigmoid edge is drawn in Fig. 5. In this paper, the sigmoid edge is defined as one edge pattern which has continuous rising or decreasing profile, but the span in its profile between different pixels might be different. Generally, the edge point should be the middle point $x_0$ in the slope of the sigmoid edge.

Fig. 18

A sigmoid edge

For the extremum rules, since the sigmoid is similar as the ramp, the first and the second rule can be proved as the same procedures as the ramp edge in the Sect. 1. Based on the same deduction, the third rule in the extremum approach can also not be guaranteed in the sigmoid edge pattern. Comparing with other three basic edge patterns, the characteristics of the sigmoid edge are not as tight as others. Based on its properties, the sigmoid edge detector is designed as,

• 1. The first-order derivative of all points in the sigmoid edge must have same sign, except the end of the sigmoid edge.

• 2. The number of minimal points in the slope is 1.

• 3. The absolute value of the difference between the onset and the end points should be larger than pre-defined threshold.

• 4. The point having the largest gradient in the slope is labeled as edges.

If the shape of a sigmoid edge is ideal as shown in the Fig. 18, the middle point should be the location of edge, which is also the point having the largest gradient in the profile. But, since the sigmoid pattern is used to improve the flexibility of the whole edge detectors in this paper, it should be defined for general cases. As the middle point of the sigmoid edge is variant for each sigmoid edge, the edge location is set as the pixel having largest gradient in its profile, which ensures that the responses of both the step and the sigmoid detectors are probably same at the same location. Therefore, this rule will reduce edge thickness and localization error. In order to make the sigmoid is versatile, the minimal points in the slope are set as 1, which is different from the ramp edge. In this condition, the middle point is labeled as an edge.