Removing mismatches for retinal image registration via multi-attribute-driven regularized mixture model
Introduction
Image registration plays an important role in computer vision, pattern recognition, and medical diagnosis [43]. The task of this registration problem is to learn correspondences between two sets of features and to update the underlying transformation. More precisely, two or more images can be aligned together and mapped into the same scene with the optimal transformation estimated from the learned correspondence. As known widely, image registration is a fundamental problem and is still challenging in the field of vision-based research.
From the perspective of the matching correspondence, image registration can be classified into two categories: area-based or feature-based methods. Intuitively, area-based methods use a specific similarity metric such as mutual information (MI) [27], [34], [40], cross correlation (CC) [12], entropy correlation coefficient (ECC) [9], and phase correlation [22], to get the correspondence between two images. Though these methods can get good registration results, they always suffer from intractable computational complexity, image distortion and illumination changes. By contrast, feature-based methods extract salient local keypoint including Harris corner [19], Scale Invariant Feature Transform (SIFT) [30], and Speed Up Robust Feature (SURF) [4], to match correspondences. Typically, sparse feature point information is used to describe the whole image, and it is easy to estimate the transformation with tractable computational complexity. In this paper, we mainly focus our study on the feature-based methods for retinal image registration.
Feature-based image registration has in common with robust feature matching which needs to construct an initial alignment between local feature points. However, the existing uncertain mismatches make the matching task more difficult. In view of the transformation estimation needing a reliable feature correspondence, we can resort to a robust mismatches removal method. For instance, Fig. 1 shows the matching problem clearly. Initial feature matches are shown in Fig. 1(b), where mismatches make the registered image give rise to an obvious ghost in Fig. 1(d), while Fig. 1 (i) shows the perfect image registration after removing mismatches. Furthermore, vector field interpolation shows the smooth vector field learning from the feature matching.
For retinal image registration, however, multi-modal image data, which is captured by different imaging sensors such as red-free and fluorescein angiography, is challenging to deal with. It is easy to extract salient feature points by SIFT or SURF descriptors for mono-modal image registration, while multi-modal image registration is difficult to construct a reliable descriptor. Though bifurcation-based methods are widely used for multi-modal retinal image registration, the vascular tree is hard to extract and segment in complex scenes such as poor-quality, occlusion by hemorrhage, and unhealthy area [1], [10].
In order to address the above problems, then we propose a novel method called multi-attribute-driven regularized mixture model (MAD-RMM). The method can be used to learn the underlying correspondence efficiently when facing mismatches. More precisely, the initial matching correspondence is always contaminated by mismatches, let us define the correct matches and mismatches to be inliers and outliers, respectively. Subsequently, we use a Gaussian distribution and a uniform distribution to model inliers and outliers, respectively, then both of them can be absorbed into a mixture model. Under this mixture model, we make use of a novel multi-attribute-driven approach to identify inliers. In the MAD-RMM, a Maximum A Posteriori (MAP) solution of the unknown parameters of the transformation can be solved by the Expectation Maximum (EM) algorithm [13] where E-step computes the responsibilities and M-step updates the transformation, thus the underlying correspondence can be recovered by computing the maximum expectation after the algorithm convergence. Moreover, we add an L2 norm regularization term to constraint the non-linear transformation preserving smooth in a reproducing kernel Hilbert space (RKHS) [2], [37] and avoiding the ill-posed problem in the optimization procedure. In many retinal image registration applications, though high-order transformation models show accurate results, we found that the linear transformation model such as affine [21], can fit most experimental cases, so we can apply the MAD-RMM algorithm with an affine model to register retinal images.
Our contribution in this paper includes the following two aspects: (1) we propose a regression method to learn the correspondence for the feature-based matching. The initial correspondence is mapped to a high dimensional space by a specific kernel method, and the proposed learning approach can identify inliers from outliers in the special feature space (RKHS). (2) we introduce a multi-attribute-driven technique which can be efficient to guide the mixture model to recover the underlying inliers.
Finally, in our previous work, we have presented the SURF-PIIFD [49] which improves the descriptor PIIFD (partial local intensity invariant feature descriptor) [10] for multi-model retinal image registration, and the SURF-PIIFD can be used to capture more reliable feature points to construct the initial correspondence.
The remainder of the paper is organized as follows. Section 2 describes background material and related work. In Section 3, we present our multi-attribute-driven regularized mixture model. In Section 4, we apply the proposed method to register retinal images. Section 5 describes the implementation details. In Section 6, we illustrate the registration performance of the proposed method on various types of retinal image pairs with comparisons to other state-of-the-art methods. In Section 7, we give a brief discussion and conclusion.
Section snippets
Related work
In this paper, we focus on digital retinal image registration which is widely used to diagnose varieties of diseases, including diabetic retinopathy, glaucoma, and age-related macular degeneration [42], [58]. As such, numerous algorithms have been presented for image registration [1], [3], [8], [10], [26], [44], [47], [49], [54], [55]. They aim to recover the underlying correspondence or find the robust estimation of transformation to align images. Here, we briefly overview these algorithms
Notation
We use the following notation in our method.
- •
Image pair. Let I1 and I2 be the fixed target image and the moving image respectively. We aim to register the moving image onto the target image in the registration problem.
- •
Point set pair. Let and be the feature point set extracted from I1 and I2 respectively, where location inform shape context descriptor and local feature descriptor D1 and D2 denotes the dimension of
Application to retinal image registration
In the MAD-RMM algorithm, a smooth mapping function from moving point set to the target point set is solved to recover the underlying correspondence and align them together. Generally, the proposed algorithm can be applied to 2D image matching with specific feature detectors and descriptors. In this paper, we focus on the retinal image registration, particularly the multi-modal data, and apply the MAD-RMM algorithm to retinal image registration.
Implementation details
Scale parameter σ2 controls the width of capture range for each Gaussian mixture model, and we initialize it as . GRBF parameter β controls the width of the Gaussian kernel, and it produces locally smooth transformation and globally translation transformation with setting small and large values, respectively. In the experiments, we set . Regularization parameter λ trades off the empirical risk and the smoothness regularization term, here we set . Outlier weight
Experiments and results
In this section, we evaluate the performance of our proposed MAD-RMM algorithm against to four state-of-the-art methods: RANSAC [16], SVR [28], CPD [38], and GMM-TPS [24]. All the experiments were performed in Matlab 2015a on a PC with a 2.5 GHz Intel i5 Core system and 8GB RAM, where all the methods’ parameters are fixed throughout these experiments.
Discussion and conclusion
Typically, the initial matches computed by local descriptors contain mismatches because of the ambiguity of the local appearance information around the keypoints. Then, an optimization procedure is applied that uses the global position information to refine the matches and eliminate outliers. Mismatch removal plays a key role in image matching and registration, and the proposed MAD-RMM is a hybrid method which combines feature matching and spatial mapping correspondence estimation. Note that
Acknowledgment
This work was partially supported by National Natural Science Foundation of China (NSFC 61103070), and the Fundamental Research Funds for The Central Universities.
References (58)
- et al.
Image registration by compression
Inf. Sci.
(2010) - et al.
A new point matching algorithm for non-rigid registration
Comput. Vis. Image Und.
(2003) Simulated annealing: Practice versus theory
Math. Comput. Model.
(1993)- et al.
Improved image registration by sparse patch-based deformation estimation
NeuroImage
(2015) - et al.
Improving accuracy and efficiency of mutual information for multi-modal retinal image registration using adaptive probability density estimation
Comput. Med. Imag. Grap.
(2013) - et al.
Non-rigid visible and infrared face registration via regularized gaussian fields criterion
Pattern Recognit.
(2015) - et al.
Regularized vector field learning with sparse approximation for mismatch removal
Pattern Recognit.
(2013) - et al.
A multiscale elastic registration scheme for retinal angiograms
Comput. Vis. Image Understand.
(2004) - et al.
Using optical imaging summary data to detect glaucoma
Ophthalmology
(2001) - et al.
A robust non-rigid point set registration method based on asymmetric gaussian representation
Comput. Vis. Image Understand.
(2015)
Robust point matching method for multimodal retinal image registration
Biomed. Signal. Process.
Sparse representation classifier for microaneurysm detection and retinal blood vessel extraction
Inf. Sci.
A low-dimensional step pattern analysis algorithm with application to multimodal retinal image registration
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Theory of reproducing kernels
Trans. AM. Math. Soc.
Surf: Speeded up robust features
Computer Vision-ECCV 2006
Shape matching and object recognition using shape contexts
IEEE Trans. Pattern Anal.
A method for registration of 3-d shapes
IEEE Trans. Pattern Anal.
Imaging vector fields using line integral convolution
Proceedings of the 20th annual conference on Computer Graphics and Interactive Techniques
A feature-based, robust, hierarchical algorithm for registering pairs of images of the curved human retina
IEEE Trans. Pattern Anal.
Hybrid retinal image registration
IEEE Trans. Inf. Technol. B.
A partial intensity invariant feature descriptor for multimodal retinal image registration
IEEE Trans. Biomed. Eng.
Maximum likelihood from incomplete data via the em algorithm
J. R. Stat. Soc. B., (1977)
Elastic registration for retinal images based on reconstructed vascular trees
IEEE Trans. Biomed. Eng.
Gaussian fields: a new criterion for 3d rigid registration
Pattern Recognit.
Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography
Commun. ACM
Genetic algorithms and machine learning
Mach. Learn.
The fast gauss transform
SIAM J. Sci. Stat. Comput.
A combined corner and edge detector.
Alvey vision conference
Cited by (19)
Learning under (1 + ϵ)-moment conditions
2020, Applied and Computational Harmonic AnalysisA-RANSAC: Adaptive random sample consensus method in multimodal retinal image registration
2018, Biomedical Signal Processing and ControlCitation Excerpt :In addition, due to the complex nature of medical images including images of retina, a large number of mismatches are created in the algorithm that ultimately lead to interference in the process of multimodal retinal image registration. A great deal of research has been done to remove mismatches, the examples of which include Multi-Attribute-Driven Regularized Mixture Model (MAD-RMM) [37], Graph Transformation Matching (GTM) [38] and RANdom Sample Consensus (RANSAC) [39]. RANSAC is a robust estimation method introduced by Fischler [39].
Gaussian field consensus: A robust nonparametric matching method for outlier rejection
2018, Pattern RecognitionCitation Excerpt :Experimental setup, results, and comparative studies are reported in Section 5, followed by some concluding remarks in Section 6. Many methods exist for outlier rejection in computer vision and pattern recognition, particularly in the field of image stitching [22,23], registration (retinal image [24,25], remote sensing image [26]). They aim to remove the false matches from the putative correspondences, i.e., recover the correct correspondence.
Fuzzy correspondences guided Gaussian mixture model for point set registration
2017, Knowledge-Based SystemsRetinal fundus image registration framework using Bayesian integration and asymmetric Gaussian mixture model
2023, International Journal of Imaging Systems and TechnologyA Pair-Metamorphosis-Decouple Synthetic Data Scheme for Color Fundus Image Registration
2022, Proceedings - IEEE International Conference on Multimedia and Expo