Corner detection based on shearlet transform and multi-directional structure tensor

https://doi.org/10.1016/j.patcog.2020.107299Get rights and content

Highlights

  • It took full advantages of shearlet to obtain structural information.

  • Improved shearlets to overcome the weakness of the current shearlets for corner detection.

  • A novel multi-directional structure tensor for making full use of structural information.

  • A new multi-scale corner measurement function for exactly refining candidate corners.

  • It yields some improvements on the current state-of-the-art in corner detection.

Abstract

Image corners have been widely used in various computer vision tasks. Current multi-scale analysis based corner detectors do not make full use of the multi-scale and multi-directional structural information. This degrades their detection accuracy and capability of refining corners. In this work, an improved shearlet transform with a flexible number of directions and a reasonable support is proposed to extract accurate multi-scale and multi-directional structural information from images. To make full use of the structural information from the improved shearlets, a novel multi-directional structure tensor is constructed for corner detection, and a multi-scale corner measurement function is proposed to remove false candidate corners. Experimental results demonstrate that the proposed corner detector performs better than existing corner and interest point detectors in terms of detection accuracy, localization accuracy, and robustness to affine transformations, illumination changes, noise, viewpoint changes, etc. It has a great potential for extension as a descriptor and for applications in computer vision tasks.

Introduction

Image corner is an important local feature in image processing, and corner detection is a critical pre-processing step for many image processing tasks such as image matching, pattern classification, motion estimation, and object tracking [1]. Moravec [2] found an obvious property in the grayscale patterns of corners and pioneered the study of corner detection. Based on Moravec’s finding, the well-known Harris corner detector [3] was proposed and a ‘corner score’ was derived for an image pixel from a two-directional structure tensor formed from image gradients. Meanwhile, other methods such as Hessian matrix [4] and Hilbert transform [5] have also been used to derive the corner score. These methods usually determine the corner score by analyzing a two-directional structure tensor constructed from the first or second-order derivatives of the image. They are simple and efficient, but sensitive to noise. Moreover, two directional derivatives cannot accurately depict the difference between edges and corners. For example, Noble [6] analyzed the principle behind the Harris corner detector and pointed out that the Harris detector can only detect ‘L’ shaped corners well.

In order to overcome this deficiency, Zhang and Sun [7] replaced the Gaussian kernel in the Harris detector [3] with a set of anisotropic Gaussian kernels to establish an eight directional structure tensor based on the first-order derivatives of an image. Meanwhile, the template-based methods [8], [9], [10], [11], [12] and contour-based methods [13], [14], [15], [16], [17], [18] have also been presented. The contour-based methods detected corners by analyzing the shape characteristics of the edges in images. Their results heavily depend on the prior segmentation and boundary tracking. The template-based methods used different types of parameterized corner templates to detect corners. Since the parameterized corner templates cannot cover all the types of corners that have different orientations and subtended angles, the performance is unsatisfactory in practical applications.

Recently, multi-scale analysis techniques have been used for corner detection. Mikolajczyk and Schmid proposed two scale invariant detectors called Harris-Laplace [19] and Hessian-Laplace [20], which used Laplacian of Gaussian to perform multi-scale analysis. Wavelets and lifting wavelets have also been used for corner detection [21], [22]. Since two-dimensional wavelets have a limited capability in capturing directional information, Gabor wavelets and Log-Gabor wavelets were used for obtaining more directional information. Gao et al. [23] proposed a corner detection method, namely LGWTSMM, in which the Log-Gabor wavelet coefficients in all directions were weighted and summed to establish a two directional structure tensor for corner detection. This method outperforms the aforementioned wavelet-based methods, but the Gabor kernel is highly non-orthogonal and can cause redundancy in the coefficients [24]. The redundancy will decrease the distinction between corners and other pixels. Moreover, the process of weighted summation of structural information from all directions will lead to loss of some useful structural information and increase the corner localization error.

Natural images contain intrinsic geometrical structures that are important for corner detection. The aforementioned wavelets are good at isolating the discontinuities at edge points, but they cannot handle equally well distributed singularities such as those along curves [25]. As a result, contourlets [26] and shearlets [27] have been introduced to capture the intrinsic geometrical structures. Duan et al. [28] employed a 3D version of the traditional shearlet transform with a dual tree structure to capture the structure information from 3D magnetic resonance images, and then, the structure tensor in the Harris corner detector was utilized to analysis the structure information for image fusion. Malafronte et al. [29] constructed a spatio-temporal interest point detector based on shearlet transform, in which the shearlet coefficients in all directions at each scale were summed. The products of the resultant values at the two finest scales were compared with a threshold for interest point detection. Duval-Poo et al. [30] proposed another interest point detection method based on shearlets. They developed a measure function by summing the shearlet coefficients in all directions at all scales for interest point detection, which has been proven to be relatively accurate in detection and highly robust to noise and scale variations. However, the traditional shearlet transform suffers from some inadequacies in corner detection including the following: (1) shearlet transform suffers from a strong side lobe effect. The local structural information at the locations where the grayscale values drastically change, such as along edges or at corners, has multiple peaks; (2) since a corner is not simply formed by straight lines, and there might be some intensity changes in several directions in its background, the structural information in two adjacent directions is not completely independent. However, in shearlet transform, there is no overlap between the shearlets of adjacent directions in the Fourier domain, making the structural information of adjacent directions independent; (3) because the mother function of shearlets has a limited support in the Fourier domain, the corresponding spatial filter has a larger envelope. This results in the problem of edge extension, which causes erroneous detections in the case of the extensions of two edges intersecting with each other; (4) a spatial filter with a larger envelope depresses the high frequency components in the image, making the detailed information less clear; and (5) in shearlets, the orientation variable is associated with the scale index. For different scales, the number of directions is different and is determined by the scale index. This leads to a great difficulty for subsequent corner detection by using the multi-scale directional information.

Duval-Poo et al. [31], [32] addressed the first inadequacy, i.e., the problem of multiple peaks, and they proposed an edge and corner detection method with a modified shearlet transform. In order to eliminate the effect of multiple peaks, they selected an anti-symmetric wavelet proposed by Mallat and Zhong [33] to replace the Meyer wavelet which is one of two generating functions of the mother shearlet, and the other details of the traditional shearlet transform remain unchanged. Based on the structural information obtained by the modified shearlets, they firstly calculated the mean of all scale shearlet coefficients in the same direction, and then the sum of all directional means weighted by a sinusoidal function was considered for corner detection. Benefited from the modified shearlets, this method yields a better performance in terms of detection accuracy and robustness to noise compared with the traditional shearlet based methods, but merging the multi-scale directional information by calculating their means especially merging information from coarse scales will greatly increase the corner localization errors.

In this work, we focus on all the inadequacies of the traditional shearlet transform and develop an improved shearlet transform with a flexible number of directions and a reasonable support, which can accurately obtain multi-scale and multi-directional structural information from images. In order to make full use of the multi-scale and multi-directional information, based on the shearlet coefficients, a multi-directional structure tensor is constructed for corner detection, and a multi-scale corner measurement function is proposed to reduce the influence of noise and suppress edges. The proposed corner detection method is compared with representative corner detectors. The obtained results show that the proposed corner detection method yields some improvements over the current state-of-the-arts in corner detection in terms of the detection accuracy, corner localization accuracy, and robustness to affine transformations, illumination changes, noise, image blurring, viewpoint changes, etc.

The remainder of this paper is organized as follows. The traditional shearlet transform is briefly reviewed in Section 2. In Section 3, the inadequacies of the traditional shearlet transform for corner detection are discussed and addressed. An improved shearlet transform is developed, and a novel corner detection and measurement algorithm is proposed. Extensive experimental results are given in Section 4. Finally, in Section 5, concluding remarks are provided.

Section snippets

Review of shearlet transform

In this section, the theory behind shearlet transform is reviewed, followed by the introduction of a discretization scheme for shearlets.

Shearlet-based corner detection

In this section, the inadequacies of the current shearlet transforms for corner detection are discussed and addressed. An improved shearlet transform is proposed. Then, based on the improved shearlet transform, a novel corner detection and measurement algorithm is presented.

Experimental results and performance analysis

Experimental assessments on the proposed corner detector were performed. The detection performance, robustness to affine transformations, illumination changes, noise, image blurring, viewpoint changes, and JPEG compression, and computational cost were evaluated and compared with representative corner detectors such as Harris [3], Hessian-Laplace [20], FAST [10], ORB [11], CF [16], ANDD [17], ACJ [18], LGWTSMM [23], SMCD [31], and Zhang and Sun's method [7].

In detection performance evaluation,

Conclusion

In this paper, an improved shearlet transform is proposed, and a novel corner detection method is developed based on the improved shearlet transform, multi-directional structure tensor, and multi-scale corner measurement function. The improved shearlet transform effectively overcomes the inadequacies in traditional shearlet transforms such as edge extensions and multiple peak responses. It can extract clear and accurate local structural information with a flexible number of directions from

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

The first author is supported by UNSW Tuition Fee Scholarship, China Scholarship Council (No. 201704910811), and CSIRO Data61 Scholarship. The authors would like to thank the anonymous reviewers and the editor for their constructive comments to further improve the quality of this paper.

Mingzhe Wang is pursuing his Ph.D. degree in the School of Computer Science and Engineering, University of New South Wales, Australia and CSIRO Data61, Australia. He received his M.S. degree from the Chinese Academy of Sciences, China in 2017 and B.S. degree from the Shaanxi Normal University, China in 2014. His research interests include computer vision, image analysis, and pattern recognition.

References (40)

  • H. Bay et al.

    Speeded-up robust features (SURF)

    Comput. Vision Image Understanding

    (2008)
  • H. Moravec

    Obstacle Avoidance and Navigation in the Real World by a Seeing Robot Rover

    Technical Report

    (1980)
  • C. Harris et al.

    A combined corner and edge detector

    Proc. of the Fourth ALVEY Vision Conference (ACV’88)

    (1988)
  • P.R. Beaudet

    Rotationally invariant image operators

    Proc. of the 4th International Joint Conference on Pattern Recognition (IJCPR’78)

    (1978)
  • W. Zhang et al.

    Corner detection using multi-directional structure tensor with multiple scales

    Int. J. Comput. Vis.

    (2020)
  • X. Chang et al.

    Corner detection based on morphological disk element

    2007 American Control Conference

    (2007)
  • E. Rosten et al.

    Fusing Points and Lines for High Performance Tracking

    10th IEEE International Conference on Computer Vision (ICCV’05)

    (2005)
  • E. Rublee et al.

    ORB: An efficient alternative to SIFT or SURF

    2011 International Conference on Computer Vision

    (2011)
  • S.M. Smith et al.

    SUSAN–a new approach to low level image processing

    Int. J. Comput. Vis.

    (1997)
  • B. Kerautret et al.

    Curvature based corner detector for discrete, noisy and multi-scale contours

    Int. J. Shape Model.

    (2008)
  • Cited by (25)

    • Complex shearlets and rotary phase congruence tensor for corner detection

      2022, Pattern Recognition
      Citation Excerpt :

      Three datasets (partly shown in Fig. 7 and Fig. 8) were used for evaluation. Dataset I, namely the Oxford Affine Database, comprises 6 image sequences formed from 36 images [13,17,18,38]. Dataset II [7,13,18] contains 1800 images transformed from 50 natural images, and they are combined into six different styles of image sequences.

    • Efficient corner detection based on corner enhancement filters

      2022, Digital Signal Processing: A Review Journal
      Citation Excerpt :

      Four datasets were considered, and parts of them are shown in Fig. 17 and Fig. 18. Dataset I, i.e., the Oxford Affine dataset, has been frequently used in the evaluation of corner detection repeatability [16,19,20,30], and it comprises 36 images that are combined into 6 different image sequences. Dataset II, used in [9,16,20], contains 1800 images that are transformed from 50 natural images.

    View all citing articles on Scopus

    Mingzhe Wang is pursuing his Ph.D. degree in the School of Computer Science and Engineering, University of New South Wales, Australia and CSIRO Data61, Australia. He received his M.S. degree from the Chinese Academy of Sciences, China in 2017 and B.S. degree from the Shaanxi Normal University, China in 2014. His research interests include computer vision, image analysis, and pattern recognition.

    Weichuan Zhang received the MS degree in signal and information processing from the Southwest Jiaotong University in China and the Ph.D. degree in signal and information processing in National Lab of Radar Signal Processing, Xidian University, China. He is a lecturer at Xi’an Polytechnic University. He currently is an academic visitor at CSIRO, Sydney, Australia. His research interests include computer vision, image analysis, and pattern recognition.

    Changming Sun received his Ph.D. degree in computer vision from Imperial College London, London, UK in 1992. He then joined CSIRO, Sydney, Australia, where he is currently a Principal Research Scientist carrying out research and working on applied projects. He is also a conjoint professor with the School of Computer Science and Engineering, University of New South Wales, Sydney. His current research interests include computer vision, image analysis, and pattern recognition. He has served on the program/organizing committees of various international conferences. He is an Associate Editor of the EURASIP Journal on Image and Video Processing. He is a member of the Australian Pattern Recognition Society.

    Arcot Sowmya received the Ph.D. degree in computer science from IIT Bombay, besides other degrees in mathematics and computer science. She is currently a Professor with the School of Computer Science and Engineering, University of New South Wales, Sydney. Her research has been applied to extraction of linear features in remotely sensed images and feature extraction, recognition, and computer aided diagnosis in medical images. Her areas of research include learning in vision for segmentation and object recognition, and embedded system design.

    View full text