Elsevier

Information Sciences

Volume 372, 1 December 2016, Pages 492-504
Information Sciences

Removing mismatches for retinal image registration via multi-attribute-driven regularized mixture model

https://doi.org/10.1016/j.ins.2016.08.041Get rights and content

Abstract

In order to address the problem of retinal image registration, this paper proposes and analyzes a novel and general matching algorithm called Multi-Attribute-Driven Regularized Mixture Model (MAD-RMM). Mismatches removal can play a key role in image registration, which refers to establish reliable matches between two point sets. Here the presented approach starts from multi-feature attributes which are used to guide the feature matching to identify inliers (correct matches) from outliers (incorrect matches), and then estimates the spatial transformation. In this paper, motivated by the problem of feature matching that the initial correspondence is always contaminated by outliers, thereby we formulate this issue as a probability deformable mixture model which consists of Gaussian components for inliers and uniform components for outliers. Moreover, the algorithm takes full advantage of using multiple attributes for better general matching performance. Here we are assuming all inliers are mapped into a high-dimensional feature space, namely reproducing kernel Hilbert space (RKHS), and the closed-form solution to the mapping function is given by the representation theorem with L2 norm regularization under the Expectation-Maximization (EM) algorithm. Finally, we evaluate the performance of the algorithm by applying it to retinal image registration on several datasets, where experimental results demonstrate that the MAD-RMM outperforms current state-of-the-art methods and shows the robustness to outliers on real retinal images.

Introduction

Image registration plays an important role in computer vision, pattern recognition, and medical diagnosis [43]. The task of this registration problem is to learn correspondences between two sets of features and to update the underlying transformation. More precisely, two or more images can be aligned together and mapped into the same scene with the optimal transformation estimated from the learned correspondence. As known widely, image registration is a fundamental problem and is still challenging in the field of vision-based research.

From the perspective of the matching correspondence, image registration can be classified into two categories: area-based or feature-based methods. Intuitively, area-based methods use a specific similarity metric such as mutual information (MI) [27], [34], [40], cross correlation (CC) [12], entropy correlation coefficient (ECC) [9], and phase correlation [22], to get the correspondence between two images. Though these methods can get good registration results, they always suffer from intractable computational complexity, image distortion and illumination changes. By contrast, feature-based methods extract salient local keypoint including Harris corner [19], Scale Invariant Feature Transform (SIFT) [30], and Speed Up Robust Feature (SURF) [4], to match correspondences. Typically, sparse feature point information is used to describe the whole image, and it is easy to estimate the transformation with tractable computational complexity. In this paper, we mainly focus our study on the feature-based methods for retinal image registration.

Feature-based image registration has in common with robust feature matching which needs to construct an initial alignment between local feature points. However, the existing uncertain mismatches make the matching task more difficult. In view of the transformation estimation needing a reliable feature correspondence, we can resort to a robust mismatches removal method. For instance, Fig. 1 shows the matching problem clearly. Initial feature matches are shown in Fig. 1(b), where mismatches make the registered image give rise to an obvious ghost in Fig. 1(d), while Fig. 1 (i) shows the perfect image registration after removing mismatches. Furthermore, vector field interpolation shows the smooth vector field learning from the feature matching.

For retinal image registration, however, multi-modal image data, which is captured by different imaging sensors such as red-free and fluorescein angiography, is challenging to deal with. It is easy to extract salient feature points by SIFT or SURF descriptors for mono-modal image registration, while multi-modal image registration is difficult to construct a reliable descriptor. Though bifurcation-based methods are widely used for multi-modal retinal image registration, the vascular tree is hard to extract and segment in complex scenes such as poor-quality, occlusion by hemorrhage, and unhealthy area [1], [10].

In order to address the above problems, then we propose a novel method called multi-attribute-driven regularized mixture model (MAD-RMM). The method can be used to learn the underlying correspondence efficiently when facing mismatches. More precisely, the initial matching correspondence is always contaminated by mismatches, let us define the correct matches and mismatches to be inliers and outliers, respectively. Subsequently, we use a Gaussian distribution and a uniform distribution to model inliers and outliers, respectively, then both of them can be absorbed into a mixture model. Under this mixture model, we make use of a novel multi-attribute-driven approach to identify inliers. In the MAD-RMM, a Maximum A Posteriori (MAP) solution of the unknown parameters of the transformation can be solved by the Expectation Maximum (EM) algorithm [13] where E-step computes the responsibilities and M-step updates the transformation, thus the underlying correspondence can be recovered by computing the maximum expectation after the algorithm convergence. Moreover, we add an L2 norm regularization term to constraint the non-linear transformation preserving smooth in a reproducing kernel Hilbert space (RKHS) [2], [37] and avoiding the ill-posed problem in the optimization procedure. In many retinal image registration applications, though high-order transformation models show accurate results, we found that the linear transformation model such as affine [21], can fit most experimental cases, so we can apply the MAD-RMM algorithm with an affine model to register retinal images.

Our contribution in this paper includes the following two aspects: (1) we propose a regression method to learn the correspondence for the feature-based matching. The initial correspondence is mapped to a high dimensional space by a specific kernel method, and the proposed learning approach can identify inliers from outliers in the special feature space (RKHS). (2) we introduce a multi-attribute-driven technique which can be efficient to guide the mixture model to recover the underlying inliers.

Finally, in our previous work, we have presented the SURF-PIIFD [49] which improves the descriptor PIIFD (partial local intensity invariant feature descriptor) [10] for multi-model retinal image registration, and the SURF-PIIFD can be used to capture more reliable feature points to construct the initial correspondence.

The remainder of the paper is organized as follows. Section 2 describes background material and related work. In Section 3, we present our multi-attribute-driven regularized mixture model. In Section 4, we apply the proposed method to register retinal images. Section 5 describes the implementation details. In Section 6, we illustrate the registration performance of the proposed method on various types of retinal image pairs with comparisons to other state-of-the-art methods. In Section 7, we give a brief discussion and conclusion.

Section snippets

Related work

In this paper, we focus on digital retinal image registration which is widely used to diagnose varieties of diseases, including diabetic retinopathy, glaucoma, and age-related macular degeneration [42], [58]. As such, numerous algorithms have been presented for image registration [1], [3], [8], [10], [26], [44], [47], [49], [54], [55]. They aim to recover the underlying correspondence or find the robust estimation of transformation to align images. Here, we briefly overview these algorithms

Notation

We use the following notation in our method.

  • Image pair. Let I1 and I2 be the fixed target image and the moving image respectively. We aim to register the moving image onto the target image in the registration problem.

  • Point set pair. Let XN×2={(xn,sn,pn)}n=1N and YM×2={(ym,tm,qm)}m=1M be the feature point set extracted from I1 and I2 respectively, where location inform xn,ymR2, shape context descriptor sn,tmRD1, and local feature descriptor pn,qmRD2, D1 and D2 denotes the dimension of

Application to retinal image registration

In the MAD-RMM algorithm, a smooth mapping function from moving point set to the target point set is solved to recover the underlying correspondence and align them together. Generally, the proposed algorithm can be applied to 2D image matching with specific feature detectors and descriptors. In this paper, we focus on the retinal image registration, particularly the multi-modal data, and apply the MAD-RMM algorithm to retinal image registration.

Implementation details

Scale parameter σ2 controls the width of capture range for each Gaussian mixture model, and we initialize it as σ2=n=1Nm=1Mxnym22MN. GRBF parameter β controls the width of the Gaussian kernel, and it produces locally smooth transformation and globally translation transformation with setting small and large values, respectively. In the experiments, we set β=3. Regularization parameter λ trades off the empirical risk and the smoothness regularization term, here we set λ=8. Outlier weight

Experiments and results

In this section, we evaluate the performance of our proposed MAD-RMM algorithm against to four state-of-the-art methods: RANSAC [16], SVR [28], CPD [38], and GMM-TPS [24]. All the experiments were performed in Matlab 2015a on a PC with a 2.5 GHz Intel i5 Core system and 8GB RAM, where all the methods’ parameters are fixed throughout these experiments.

Discussion and conclusion

Typically, the initial matches computed by local descriptors contain mismatches because of the ambiguity of the local appearance information around the keypoints. Then, an optimization procedure is applied that uses the global position information to refine the matches and eliminate outliers. Mismatch removal plays a key role in image matching and registration, and the proposed MAD-RMM is a hybrid method which combines feature matching and spatial mapping correspondence estimation. Note that

Acknowledgment

This work was partially supported by National Natural Science Foundation of China (NSFC 61103070), and the Fundamental Research Funds for The Central Universities.

References (58)

  • WangG. et al.

    Robust point matching method for multimodal retinal image registration

    Biomed. Signal. Process.

    (2015)
  • ZhangB. et al.

    Sparse representation classifier for microaneurysm detection and retinal blood vessel extraction

    Inf. Sci.

    (2012)
  • J. Addison Lee et al.

    A low-dimensional step pattern analysis algorithm with application to multimodal retinal image registration

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2015)
  • N. Aronszajn

    Theory of reproducing kernels

    Trans. AM. Math. Soc.

    (1950)
  • H. Bay et al.

    Surf: Speeded up robust features

    Computer Vision-ECCV 2006

    (2006)
  • S. Belongie et al.

    Shape matching and object recognition using shape contexts

    IEEE Trans. Pattern Anal.

    (2002)
  • P.J. Besl et al.

    A method for registration of 3-d shapes

    IEEE Trans. Pattern Anal.

    (1992)
  • B. Cabral et al.

    Imaging vector fields using line integral convolution

    Proceedings of the 20th annual conference on Computer Graphics and Interactive Techniques

    (1993)
  • A. Can et al.

    A feature-based, robust, hierarchical algorithm for registering pairs of images of the curved human retina

    IEEE Trans. Pattern Anal.

    (2002)
  • T. Chanwimaluang et al.

    Hybrid retinal image registration

    IEEE Trans. Inf. Technol. B.

    (2006)
  • ChenJ. et al.

    A partial intensity invariant feature descriptor for multimodal retinal image registration

    IEEE Trans. Biomed. Eng.

    (2010)
  • A.V. Cideciyan, Registration of ocular fundus images, IEEE Eng. Med. Biol. 14(1)...
  • A.P. Dempster et al.

    Maximum likelihood from incomplete data via the em algorithm

    J. R. Stat. Soc. B., (1977)

    (1977)
  • B. Fang et al.

    Elastic registration for retinal images based on reconstructed vascular trees

    IEEE Trans. Biomed. Eng.

    (2006)
  • B.A.M.A. Faysal Boughorbel

    Gaussian fields: a new criterion for 3d rigid registration

    Pattern Recognit.

    (2004)
  • M.A. Fischler et al.

    Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography

    Commun. ACM

    (1981)
  • D.E. Goldberg et al.

    Genetic algorithms and machine learning

    Mach. Learn.

    (1988)
  • L. Greengard et al.

    The fast gauss transform

    SIAM J. Sci. Stat. Comput.

    (1991)
  • C. Harris et al.

    A combined corner and edge detector.

    Alvey vision conference

    (1988)
  • Cited by (19)

    • Learning under (1 + ϵ)-moment conditions

      2020, Applied and Computational Harmonic Analysis
    • A-RANSAC: Adaptive random sample consensus method in multimodal retinal image registration

      2018, Biomedical Signal Processing and Control
      Citation Excerpt :

      In addition, due to the complex nature of medical images including images of retina, a large number of mismatches are created in the algorithm that ultimately lead to interference in the process of multimodal retinal image registration. A great deal of research has been done to remove mismatches, the examples of which include Multi-Attribute-Driven Regularized Mixture Model (MAD-RMM) [37], Graph Transformation Matching (GTM) [38] and RANdom Sample Consensus (RANSAC) [39]. RANSAC is a robust estimation method introduced by Fischler [39].

    • Gaussian field consensus: A robust nonparametric matching method for outlier rejection

      2018, Pattern Recognition
      Citation Excerpt :

      Experimental setup, results, and comparative studies are reported in Section 5, followed by some concluding remarks in Section 6. Many methods exist for outlier rejection in computer vision and pattern recognition, particularly in the field of image stitching [22,23], registration (retinal image [24,25], remote sensing image [26]). They aim to remove the false matches from the putative correspondences, i.e., recover the correct correspondence.

    • A Pair-Metamorphosis-Decouple Synthetic Data Scheme for Color Fundus Image Registration

      2022, Proceedings - IEEE International Conference on Multimedia and Expo
    View all citing articles on Scopus
    View full text