Keywords

1 Introduction

In vivo imaging is widely used for studying cell behavior in living animals because unlike to Ex vivo imaging it can replicate the physiological environment. Specifically, in the atherosclerosis study, blood flow is vital to the process of cell migration into the plaque, and ex vivo imaging hardly enables to replicate this process. However, significant cardiac and respiratory motion of the living subject, occasional spells of defocus, drifts in the field of view often spawns substantial subset of poor quality images in the long microscopy in vivo image sequence [14]. Poor quality images may propagate the error during the registration process conducted serially along the natural temporal order of this sequence. Motion artifacts associated with intravital microscopy imaging can be reduced by triggering image acquisition based on the animal heartbeat [6]. This cardiac triggering guarantees that each frame is captured at the same point in the cardiac cycle. However, some visible artifacts cannot be removed due to imprecise triggering, cardic, respiratory, and other mouse movements. Residual motion artifacts can escalate cell position and velocity error. However, cell shape and velocity plays an important role defining leukocyte behavior.

One of the long-standing challenge in in vivo image analysis is to develop the automatic image registration algorithm specially for long time sequenced images where changes in focal plane and illumination intensity, combined with inherent noise lead to the standard registration techniques a difficult task to automate the registration process. On the other hand, manual or semi-automatic registration specially for longer sequences is an extremely tedious task which demands for automation. In this study, we developed and implemented an optimal sequence optimization algorithm for microscopy image registration process. Towards achieving this goal, we first reorder the natural temporal image sequence using minimum weighted spanning tree (MST) [2]. The intuition is that MST sends back all the poor quality images at the end of the sequence which enforces to register the good images at first and the poor quality images at the end and thus mitigate error propagation when the registration is performed serially. In addition, the algorithm selects the optimal anchor image automatically and thus avoids the user interaction during the longer registration process. It has only two tuning parameters, graph width and number of clusters in the graph, making it more user friendly than many existing techniques. Experimental results show that the registration performance is robust to changes of graph width and the number of clusters in the graph. We conducted our experiment on in-vivo microscopy image sequences dedicated for the study of atherosclerosis [5]. Experimental results demonstrate that we outperform competitive methods.

2 Minimum Spanning Tree (MST) Based Clustering Algorithm for Deformable Registration

The algorithm for 2D and 3D deformable registration as well as automatic anchor selection are illustrated in Algorithms 1, 2 and 3 respectively.

Fig. 1.
figure 1

(a) A time-sequenced set of nine images. Shaded circles represent poor quality images. (b) A graph constructed on the image sequence. (c) An example of clustering minimum weighted spanning tree.

Full size image

2D deformable image registration. Our MST based clustering algorithm for 2D deformable registration consists of three sequential major steps:

Step 1: MST based graph construction and clustering. We first construct a graph for the input image sequence as shown in Fig. 1(a) and (b). Then we construct a minimum weighted spanning tree (MST) [2] as shown in Fig. 1(c).

figure a

The edge weights for the graph are determined as follows:

$$ W(I_i, I_j) = {\left\{ \begin{array}{ll} ||I_i-R(I_i, I_j)|| &{} \, \text {if } i \ne j~ \text {and}~|i-j| \le \delta \\ \infty , &{} \, \text {otherwise}, \\ \end{array}\right. } $$

where the function R(X, Y) represents the registered source image Y with respect to the target image X. The norm (||.||) represents the sum of absolute differences of pixel values between two images. We used pixel wise sum of square difference (SSD) because it is simple and straightforward to compute. Other metrics for similarity measures such as mutual information (MI), Residual Complexity (RC), Correlation Coefficient, Sum of Absolute Difference (SAD), low or high level features such texture, entropy, sift, could be also used. The value of the parameter \(\delta \) (maximum degree of a graph) controls the sparsity of the graph. A smaller value of \(\delta \) illustrates greater sparsity. For a complete graph, \(\delta = n-1\). The computational complexity of MST-based registration is \(O(\delta n)\), while that for adaptive template matching based registration is O(n). As the graph is constructed, a clustering algorithm is performed on MST which stratifies the MST into M groups as shown in Fig. 1(c). Users can select the value of M or can be chosen automatically as well [12].

figure b

Step 2: Automatic anchor selection. Unlike available methods which pick either first or middle image randomly as an anchor image, we utilize an optimal anchor image selection algorithm developed in [1] through an iterative assessment of image entropy and Mean Square Error (MSE) during the registration process as demonstrated in Algorithm 2. Structural similarity measure is widely used for registration purposes. However, due to distortions, structural similarity cannot always be maintained even for consecutive slices. In contrast, MSE is a popular tool to check whether the slices are distorted or not. High MSE values indicate a high probability of distorted and noisy slices while low MSE values indicate strong similarity between consecutive images. However, a subsequence (specifically initial and end slices) from long time sequenced slices often contain little information due to the topology of the animal organs and MSE performs poorly to choose the anchor in this situation. To avoid selecting these slices, similar to [1], we incorporate entropy information along with MSE for anchor selection. Entropy measures the mutual information and high mutual information in consecutive slices illustrate high similarity and vice versa.

Step 3: Two-stage deformable image registration. Once the anchor is selected, we implement a standard deformable registration algorithm for each group with respect to the associated anchor image. For deformable registration, we implement diffeomorphic image registration [13] and for rigid registration, we implement MISTICA [8]. Unlike single stage deformable registration, we perform a two-stage deformable registration: first with respect to the local anchor (anchor within the group images) and then with respect to the global anchor to incorporate large distortions incrementally. Sharpmean [16] only considers intragroup registration and overlooked the intergroup registration and it performs poorly when the sequences contain poor quality images and a lot of variations across the group. We first developed a two-stage registration method where in the first stage we register with respect to group mean and then in the final stage we register with respect to the global mean. Two stage registration method allows both bottom up (intra group registration) and top down (inter group registration) traversal of the tree that can handle intergroup registration error specially in the presence of poor quality shorter sequences and larger variations available in the longer image sequences; We also offer a solution for selecting image sequences from a longer 3D image sequences where a predominant interclass variations exist.

figure c

3D deformable image registration. 3D deformable registration algorithm consists of following three steps as demonstrated in Algorithm 3: (i) first we implement stackwise MST based 2D deformable registration algorithm (among the time sequenced images at each z-location separately) and obtain the anchor image for each z-location using Algorithm 2; (ii) then we implement the MST based 2D deformable registration algorithm among the anchor images selected from each z-location; (iii) Finally we implement stackwise 2D diffeomorphic registration [13] at each z-location with respect to the associated anchor image.

Efficacies of the proposed algorithm

  1. (a)

    Mitigate the registration error propagation. MST based graph clustering algorithm decomposes the original image sequence into smaller sub-sequences and these smaller sub-sequences contain poor quality images toward their end. Thus, registration error propagation is minimized. Simple template matching method lies into one extreme in that it subdivides the input sequence into the largest number sub-sequences and avoids the error propagation due to poor quality images. The anchor image in simple template matching cannot adapt to the illumination changes and drifts for long image sequences. The sequential methods, such as StackReg [10], lies on the other extreme end in that it divides the input sequence into at most two sub-sequences. So, they can handle the drift and change of illumination; however, they fail to handle occurrences of poor quality images. Our MST based graph clustering algorithm achieves a balance between these two extremities.

  2. (b)

    Incorporate larger drift. For a longer sequence, normally the topology of the anchor image is quite different from many images located far away from it due to larger drift and change of illumination. In two-stage registration, first images are registered with the local anchor, the topology of which is closer to the candidate images than the global anchor image. Then images are registered with the global anchor image. Thus in the two-stage registration, the amount of total drifts and distortions present in the sequence are divided into several parts and each part is encountered separately through local registration conducted within the group. Incremental adjustment of the two-stage deformable registration towards large distortion of the structures and artifacts tends to avoid the failure of traditional single-stage deformable registration for larger drift.

  3. (c)

    Automatic optimal anchor selection. A major issue regarding image registration involves user interaction to choose the optimal anchor image. A number of the existing algorithms require the user to choose an anchor image. Our toy example illustrates that suitable anchor image can mitigate the registration error propagation. For example, the choice of 2 and 7 over 3 and 6 for group 1 and 2 respectively in Fig. 1(c) would most likely yield a better registration. In practice, browsing through images and looking for the anchor in longer image sequences can be tedious and error-prone. We adopt an automatic anchor selection algorithm in our method as described in Algorithm 2.

Fig. 2.
figure 2

Two rows from top to the bottom illustrate two datasets (a) and (b), respectively for both 2D and 3D datasets containing the multiphoton microscopy images of the artery wall to monitor the monocyte patrolling causing endothelial damage.

Full size image

3 Experimental Results and Discussions

We conducted the experiment on four microscopy datasets (two datasets each of both 2D and 3D) of mouse arteries used in atherosclerosis study as shown in Fig. 2. Figure 2 shows the first and the last images (from left to right column) of these two sequences which are denoted as (a) and (b) (from top to bottom row). We compared our MST-based two-stage deformable registration method with three well-known rigid image stack registration methods such as, StackReg [10], time-lapse image registration or TurboReg [11], MST based registration method, MISTICA [8], and three deformable registration methods, MIRT [7], elastic registration method [9], diffeomorphic demons [13] for both 2D and 3D. We used the optimal value of \(\delta = 6\), (reported in the paper [8]) for both MISTICA and our method denoted as MSTClust2D and MSTClust3D for 2D and 3D respectively. We choose the value of the number of clusters in the graph as 6 for our method through cross validation [3]. The registration function R(X, Y) for all the methods used translational motion. Since data are collected in vivo, it is practically infeasible to have ground truth registration data for the sequences.

Fig. 3.
figure 3

SSIM for different registration methods

Full size image

We used structural similarity index (SSIM) [15], which is a popular quality metric to compare registration algorithms. The value of SSIM lies between −1 and 1. Higher SSIM values demonstrate better registration between two images. The mean and SD of all the competing methods and our MST-based clustering method (MSTClust2D and MSTClust3D) for two datasets (a) and (b) of 2D and 3D are illustrated in Fig. 3. From these comparisons, it is evident that our MST-based clustering algorithm outperforms other methods for two image sequences of significant length for both 2D and 3D datasets. In addition, our method is a generalized sequence optimization technique and it can be exploited to any standard method for time sequenced image registration. Experimental results demonstrate that proposed method uses MISTICA and Diffeomorphic demons and performs better than both of these methods. Instead of MISTICA and Diffeomorphic demons, user can incorporate other suitable 2D image registration algorithms such as rigid image stack registration, time lapse image registration, elastic registration, residual complexity based deformable registration into our method for different applications. No other registration methods do not consider reordering of the sequence. However, these techniques perform significantly better than its own performance (along the order in which images were captured) if it runs along the optimal sequence selected by our method.

4 Conclusions and Future Works

We have first offered a generalized framework for selecting optimal sequence from deformable registration of both 2D and 3D datasets of long time sequenced in-vivo microscopy images containing noises such as occurrences of poor quality images, including noisy or low-intensity data, as well as images with distortion due to cardiac and respiratory motion. The intuition is that suitable re-ordering of the natural temporal order of image sequences in which they were captured sends back the poor quality images at the end of the sequence. Sequential registration algorithms register the poor quality images at the end of the registration process and thus reduce the error propagation. We achieve this by developing a computationally efficient Minimum Spanning Tree (MST)-based graph clustering algorithm. Proposed two-stage deformable registration algorithm incrementally accommodates larger drifts and distortion and outperforms conventional one stage deformable registration. Experimental results obtains state-of-the-art performance in both 2D and 3D datasets of atherosclerosis study. In future, we intend to implement the proposed algorithm in other disease studies.