A Method on Recognizing Transmission Line Structure Based on Multi-level Perception

Abstract

The structure of transmission line can be recognized by processing the images captured from unmanned aerial vehicle (UAV) power inspection. That can be applied to vision based UAV intelligent inspection and the further analysis of the fault diagnosis of transmission lines. For that, a multi-level perception-based method of transmission line structure recognition is proposed. Firstly, the extracted line segments are split based on key points and then merged based on Gestalt perception theory for getting a relatively complete and independent local contour. Next the area of parallel line segments and symmetrical and crossing line segments are perceived, and then a position restraint mechanism of transmission line structure is built for the preliminary recognition. Finally, the local contour feature is used for the further recognition. In the experiment, the false rate and the missed rate of the method are verified to be lower.

1 Introduction

With the rapid development of UAVs technology in recent years, they are applied not only to the construction of geographic information system but also to the inspection of transmission line. Currently, UAVs are actively used by the China State Grid Corporation for improving the efficiency of visual transmission line inspection.

The research on transmission line structure is focused on the recognition of the components in the transmission line, and the image features such as shape points and characteristic points are important to the recognition of these artificial objects. SIFT feature matching [1] is a common matching method based on local feature, and the method is robust to image rotation, translation and scale transformation. But extraction error and matching error of the feature point would be high and the calculation of related data processing would be complex in natural scene of transmission lines. The shape recognition method like Fourier Descriptor [2] can reduce the influence of background noise and the sensitivity of boundary variation, Chen [3] used independent edge segments and closed contours searched by Generic Edge Token (GET) graph for object recognition. These shape recognition methods are only applicable to closed contour, but the extracted contours from the object in natural scene of transmission lines are usually incomplete because they are affected by background interference and mutual occlusion of the structure itself. By the study of human visual characteristics [4], this thesis can accurately identify the object according to the local contour, which shows that the local contour feature can be used to describe the object accurately. Ferrari [5] used the edge chain code to cluster K adjacent edge segments (KAS) as the local contour feature of the object. Ban [6] and Zhu [7] analyze the contour feature consisting of 2 or 3 adjacent edge segments of the object, and the semantic structure model of the contour feature is defined according to the angle between adjacent segments, segments’ length and line vectors. However, these methods mainly focus on the single and significant objects (such as the insulator of transmission lines), and ignores the study on the overall structure of transmission lines. So these methods are easily affected by complex background texture and light, and more regions with similar features will be detected in the background.

The main difficulties of this paper as follows: (1) Transmission line is a kind of 3D hollow line structure, and there are a lot of occlusions in different shooting angles, so it’s hard to extract the complete contour for the recognition; (2) The images taken by UAVs are greatly influenced by the background texture and the light changing, and there are many regions with similar features in the background, so it will easily cause an erroneous judgement.

2 Acquisition of Contour Segments of Transmission Line

The major objects of the recognition of transmission line structure are installed parts, and the problems mentioned above also need to be solved. An inference mechanism based on multi-level perception to recognize the structure of transmission line is proposed, and it is shown in Fig. 1.

Fig. 1.

The flow diagram of algorithm. (a) The low-level for image processing. (b) The middle-level for image analysis. (c) The high-level for semantic description of image

At the bottom of image processing (Fig. 1(a)), the crossing gradient template [8] with a variable width is adopted to extract line segments. Considering that the object contour may be fused with the background, the splitting method is applied to the contour segments. Due to the influence of the light and occlusions, the extracted line segments may be discontinuous, then the line segments should be merged.

In the middle of image analysis (Fig. 1(b)), the significant characteristics are a major concern, approximate parallelism and approximate symmetrical cross. The power lines are a kind of parallel line structure, and the tower is an approximately symmetrical crossing line structure, a perception method based on blocking and clustering is pro-posed for perceiving these parallel and symmetrical crossing structures, and the region of the object to be recognized can be determined by both characteristic.

A high-level semantic description of image (Fig. 1(c)) is proposed to approach visual sense of human being, position constraint and local contour constraint. Analysis of the installation position of the object, a position constraint mechanism is built according to the connection of the object and power lines. Through the perception of the position constraint, the region of the object will be limited to a small area. Then combining the local contour features of the object, the structure of transmission line can be recognized semantically.

The specific recognition method is introduced in the Sects. 3, 4 and 5, the experiments and the conclusions given in Sects. 6 and 7 respectively.

3 Acquisition of Contour Segments of Transmission Line

3.1 Splitting Based on Key Points

Due to the influent of light, background texture and occlusions, the extracted line segment may be the segment with inflection point. The inflection point formed by the connection of the object contour with the background line or other parts, so the extracted segment should be divided into two segments. The detailed method of searching an inflection point as follows:

• 1. Each line segment are divided into SegNumber segments, and slopes of those are calculate by least square method, represented as: $\theta_{1} ,\,\theta_{2} ,\, \ldots \theta_{s}$.

• 2. The slope angle difference between two adjacent lines is calculate, $\Delta \theta_{1} ,\,\Delta \theta_{2} ,\, \ldots \Delta \theta_{s - 1}$.

• 3. The inflection point satisfies the following conditions:

$$ abs(\Delta \theta_{i} ) > = angle\_threshold $$

(1)

3.2 Merging Based on Gestalt Theory

The approximation, the co-linearity and the continuity of Gestalt perception law [9, 10] is applied in line segments merging. Therein Gestalt means when parts identified individually have different characteristics to the whole. For getting more complete contour segments of transmission line structure, the description of segments merging based on Gestalt perception law shown in Fig. 2.

Fig. 2.

Gestalt perception law calculation description

where L1 and L2 represent the lengths of two adjacent segments respectively, θ ₁ and θ ₂ corresponding to inclination angle of the segments.

The distance L between two adjacent segments is used to describe the approximation. If L satisfied the formula (2), it would comply with the approximation.

$$ L \le 2 \times \hbox{min} (L1,L2) $$

(2)

If the angle variation of two adjacent segments satisfied the formula (3), it would comply with the continuity.

$$ abs\left( {\theta_{ 1} - \theta_{ 2} } \right) \le T_{\theta } $$

(3)

The co-linearity is used to determine whether two segments was parallel or not, the vertical distance dist from the end of one segment to the other segment is a decisive factor to describe the co-linearity, as formula (4) shown.

$$ dist = \frac{{abs(ss_{2} \times l1\_endx - l1\_endy + sl_{2} )}}{{\sqrt {ss_{2}^{2} + 1} }} $$

(4)

where ss2, sl2 are the slope and intercept of starting end in segment 2. l1_endx, l1_endy are the coordinates of the endpoint in segment 1. If the formula (4) was satisfied, it would be a co-linearity judgment.

$$ dist \le k*\hbox{max} (lw1,lw2) $$

(5)

Among them, lw1 and lw2 were line width of segment1 and segment2.

4 Perception of Significant Characteristics of Transmission Line

4.1 Perception of Parallel Structure and Symmetrical Crossing Structure

In order to detect the near parallel structure, the UAV image is divided into multiples of 8 blocks along the vertical direction, and line segments in each block are classified by the slope and the intercept. The result of the classification is the segments group which satisfies a certain condition of distance and angle, the parallel segments group. Then parallel segments group of adjacent blocks composed of a new parallel segments groups, according to the same endpoint location and the small angle difference until all the same parallel segments groups together.

Each block was divided into four equal parts horizontally for the perception of symmetrical crossing structure. Statistics is done for the number of different direction line segments in each part, including the number of horizontal line segments (−15° ~ 15°) hnumi; the number of upward line segments (15° ~ 75°) upnumi; the number of downward line segments (−75° ~ −15°) downnumi; the number of vertical line segments (>75°or < −75°) vnumi. If upnumi and downnumi were both higher than a present threshold, the part would belong to the symmetrical crossing part. The total number of the symmetrical crossing part is used to determine whether there exists a tower area in the image, and the region of the target object can be preliminarily determined by the perception of the tower area.

4.2 Perception of Connecting Structure and Attachment Structure

Context [5] is often used to describe the position relation between different structures in the scene. According to the structure of the transmission line, there is a position constraint of connection and attachment between the small parts and power lines, as shown in Fig. 3.

Fig. 3.

Relative position of drainage line and power line in the strain tower. (a) The connection structure between power lines and drainage lines. (b) The connection structure between power lines. (c) The attachment structure between power lines.

One end of the insulator is connected with power lines, and the other end is connected with the tower. When power lines pass through the intermediate support, a strain formed by the power lines connected with the insulator, and a polyline structure presented. When power lines pass through the strain tower, the power lines are connected by a downward convex drainage line. Based on the calculation of the distance and the angle between the ends of adjacent parallel segment groups, a position constraint of connection can be established, and the insulator can be recognized around the area of the connection structure. The hammer and the spacer are installed on power lines, and an attachment is produced in the breakpoint area between the adjacent segment groups and these small parts in the image. According to the attachment structure, the recognition of the installed parts can be finished by searching along the direction of power lines.

5 The Recognition of Transmission Line Structure

5.1 The Recognition of Insulator

From the structure of the insulator, it consists of circular shaped chips joined together. Parallel arc structure is a significant feature of insulator in UAV image. Firstly, parallel line segments are extracted by different directions, and then the contour features are calculated. This coupled with the installation position of insulator in transmission line, the restraint mechanism of recognition of insulator is set up. The contour features calculation of insulator is shown in Fig. 4, the perception of parallel shape.

Fig. 4.

Calculation of insulator parallelism

As for the parallel segments perception calculation of insulator, central points of line segments are kept in line, segments’ length and the distance between adjacent segments keeping in a certain range. Orientation angle of line segment L1 and that of line segment L2 are separately represented as θ₁ and θ₂. If θ₁ and θ₂ satisfied formula (6), L1 would parallel to L2.

$$ abs(\theta_{1} - \theta_{2} ) < = 2T_{\theta } $$

(6)

The orientation angle of the connecting between two center points of L1 and L2 is dθ ₁ and that of the connecting between center points of L2 and L3 is dθ ₂ . If these angles satisfied the following Eq. (7), L1, L2 and L3 would be in alignment.

$$ abs(d\theta_{1} - d\theta_{2} ) \le 2.5*T_{\theta } $$

(7)

where the parameter T _θ is a threshold for human visual perception of parallel lines.

The length of L1 and L2 are L1_len and L2_len respectively. If lengths satisfied the following Eq. (8), lengths would be in consistency.

$$ abs\left( {L1\_len - \, L2\_len} \right) < = min\left( {L1\_len, \, L2\_len} \right)/3 $$

(8)

Considering the distance between chips of insulator less than the diameter of the chip, the distance between central points of L1 and L2 is less than the average length of L1 and L2.

$$ dist < = avg\left( {L1\_len, \, L2\_len} \right) $$

(9)

5.2 The Recognition of Insulator

According to prior knowledge of the object to be recognized, two adjacent segments (2AS) or three adjacent segments (3AS) as the local contour feature for the object recognition, the angle and relative scale between adjacent segments are used to define the semantic model of 2AS, as shown in Fig. 5.

Fig. 5.

2AS segment diagram. (a) Schematic diagram of 2AS. (b) 2AS rotation angle relationship diagram.

The intersection of two approximate adjacent line segments is taken as the initial point to calculate vectors of each line segment, $\mathop {BA}\limits^{ \to } = r_{1} \left( {x_{1} ,y_{1} } \right)$ and $\mathop {BC}\limits^{ \to } = r_{2} \left( {x_{2} ,y_{2} } \right)$. And the angle between two vectors is also calculated. And the longest line segment is selected as the first segment to match and the length of the segment is as the normalization factor. The 2AS semantic modal can be described as formula (10).

$$ \left( {\frac{{r_{1} }}{{L_{1} }},\frac{{r_{2} }}{{L_{1} }},\alpha ,\;1,\;\frac{{L_{2} }}{{L_{1} }}} \right)\left( {\frac{{r_{1} }}{{L_{1} }},\frac{{r_{2} }}{{L_{1} }},\alpha ,\;1,\;\frac{{L_{2} }}{{L_{1} }}} \right) $$

(10)

Local contour features (marked as a and b) of 2AS were matched from the following three aspects.

• (1) The angle differences between a and b should satisfy the condition 12. θ_thr is a given threshold.

  $$ abs\left( {\alpha^{a} - \alpha^{b} } \right) \le \theta_{thr} $$

  (11)

• (2) The length differences of a and b should satisfy the condition 13.

  $$ L\_thr1 \le \left( {{{L_{2}^{a} } \mathord{\left/ {\vphantom {{L_{2}^{a} } {L_{1}^{a} }}} \right. \kern-0pt} {L_{1}^{a} }} \times {{L_{1}^{b} } \mathord{\left/ {\vphantom {{L_{1}^{b} } {L_{2}^{b} }}} \right. \kern-0pt} {L_{2}^{b} }}} \right) \le L\_thr2 $$

  (12)

• (3) the slope angle difference between two adjacent lines is calculate, $\Delta \theta_{1} ,\,\Delta \theta_{2} ,\, \ldots \Delta \theta_{s - 1}$.

  $$ abs\left( {\arccos \left( {\frac{{r_{1}^{a} \cdot r_{1}^{b} }}{{\left| {r_{1}^{a} } \right| \cdot \left| {r_{1}^{b} } \right|}}} \right) - \arccos \left( {\frac{{r_{2}^{a} \cdot r_{2}^{b} }}{{\left| {r_{2}^{a} } \right| \cdot \left| {r_{2}^{b} } \right|}}} \right)} \right) < \theta_{thr} $$

  (13)

Figures 6 and 7 were descriptions of local contour features of the spacer and the hammer. Local contour features of these shaped components were grouped by coding and such groups were adopted to travel the 2AS and 3AS of the clustering region traversal, and then the identification of hammer and spacer was realized.

Fig. 6.

The description of local contour features of hammer. (a) The shape of hamper. (b) The local contour coding label of hamper. (c) The local contour combination of hamper.

Fig. 7.

The description of local contour features of spacer. (a) The shape of spacer. (b) The local contour coding label of spacer. (c) The local contour combination of spacer.

6 Experiment Result

The computer used in the experiment is Intel(R) Core(TM) i5-3470 CPU 3.20 GHz, 4 GB RAM, NVIDIA Geforce GTS 450, and the operating system is Microsoft Windows Window 7 Professional.

The recognition result of insulator is shown in Fig. 8, which is done largely through position constraint and local contour feature. Figure 1(a) is the original image, Fig. 1(b) showing the final result.

Fig. 8.

A Recognition of structure of intermediate support. (a) The original image with the insulator. (b) The image with the recognized insulator.

Local contour feature is adopted in the method of Zhang [11], but the method is not suitable the case of that the insulator without obvious local contour feature and its local contour feature similar to background textures. The proposed method in this paper has made some improvements, the position constraint of the connection, the region of insulator can be inferred, as shown in Fig. 9. Statistics of the recognition result of the 2000 insulators are shown in Table 1, C_Num, F_Num, M_Num, C_Rate, F_Rate and M_Rate represent the correct recognition number, the false recognition number, the missed recognition number, the correct recognition rate, the false recognition rate and the missed recognition rate. From this table, the false rate and the missed rate of the method has been improved evidently. At last, the average running time of each image is calculated, 1.665 s, 1.689 s correspond to Zhang’s method, and proposed method, respectively. By comparison, there is little difference in running time.

Fig. 9.

Recognition of structure. (a) The position constraint of the connection. (b) The recognition of the insulator.

Table 1. Statics of the recognition of the insulator.

The recognition results of the 2000 hammers are shown in Fig. 10. For the hammer recognition method of Zhu [7], with no idea for reducing the influence of the similar structure, a comparison is made between Zhu’s and the proposed in Table 2.

Fig. 10.

A The recognition results of hamper

Table 2. Statics of the recognition of the hammer.

At last, the average running time of each image is calculated, 1.015 s, 0.934 s correspond to Zhu’s method, and proposed method, respectively. By comparison of running time, our proposed method is a little below.

7 Conclusions

A multi-level perception-based method of transmission line structure recognition is proposed. The position mechanism of the connection and the local contour feature are used to build the semantic knowledge model of the objects recognition, and then the objects in transmission line are recognized semantically and reliably. Finally, the experimental results show that the recognition effect has been improved by the proposed method. And the proposed algorithm will be optimized under GPU in the future work for timeliness.