1 Introduction

Phase contrast microscope has been widely used to visualize live cells without staining them [1]. It yields the image intensity as a function of specimen’s optical path length. If the phase telescope (or Bertrand lens) and substage condenser in the optics are not properly aligned, the phase contrast image will be blurry [2] (appears to be out of focus, as shown in Fig. 1(a)), which will make the specimens obscure and it will be very difficult to detect the edges of the specimens and the location of the nuclei. Moreover, the surrounding medium and the specimens may have very similar optical path lengths, which will also increase the difficulty of detecting the edges and the nuclei (e.g., cell A in Fig. 1(b)).

The two problems motivate us to think whether we can refocus a phase contrast microscopy image from two perspectives: (1) Optics: we want to estimate a blur kernel to refocus the blurred phase contrast image due to the misaligned optics; and (2) Specimens: we want to enhance the contrast between the specimens and the background such that the background is smoothed with uniform intensity values and the image contents are focused on specimen details only.

Fig. 1.
figure 1

(a, b) Defocused phase contrast microscopy images; (c, d) our refocused images.

1.1 Related Work

The first perspective of our refocusing problem is related to single image blind refocusing problem. In the past decade, many single image blind refocusing methods have been proposed. Zhang and Cham correct the blurry edges to sharp ones with the aid of a parametric edge model and then render this cue as a local prior to ensure the sharpness of the latent image [3]. Shan et al. present a unified probabilistic model of both blur kernel estimation and unblurred image restoration, which includes a model of the spatial randomness of noise in the blurred image and a local smoothness prior that reduces ringing artifacts [4]. Pan et al. propose an \(\ell _{0}\)-regularized prior based on intensity and gradient for single text image deblurring [5]. Pan et al. present a blind single image deblurring method based on the dark channel prior [6].

These general blind deblurring methods are proposed to process natural images and do not take any special image formation process into consideration. Yin et al. derive a linear imaging model for phase contrast microscopy and try to restore the artifact-free phase contrast image with a mathematically-derived Point Spread Function (PSF) [7]. Su et al. revisit the phase contrast imaging model, and propose a novel restoration algorithm which can restore phase contrast images with various phase retardations [8]. However, these two phase contrast image restoration methods remove cell details dramatically.

1.2 Our Proposal

In this paper, we investigate a refocusing algorithm to refocus phase contrast microscopy images (e.g., Fig. 1(c, d)). First we estimate a blur kernel for the defocused image and refocus it from the optics perspective. Then, considering the optical properties of the phase contrast imaging system, we propose a novel optimization method to further refocus the microscopy image on specimen details while smoothing the background, i.e., the contrast between specimens and background is enhanced, which provides a better visibility on specimens.

2 Methodology

In this section, we describe the two steps of our refocusing algorithm: (1) refocusing a blurry phase contrast microscopy image caused by misaligned optics; and (2) further refocusing the image on specimens only.

2.1 Refocusing Images from the Optics Perspective

Problem Formulation: An image blurring process can be modeled as the convolution of the focused image F with the blur kernel (or PSF) h,

$$\begin{aligned} I = F \otimes h + n, \end{aligned}$$
(1)

where I is the defocused image, n represents the noise, and \(\otimes \) denotes the convolution operator. The defocused image I here is produced from the misaligned optics components of the phase contrast microscope, we can not use the PSF in [7, 8], which is derived based on the well-aligned phase contrast microscope.

Estimating h: Obtaining the focused image F by solving Eq. (1) is ill-posed, since both the blur kernel h and the latent focused image F are unknown. In order to estimate the blur kernel and get the latent focused image, we propose to solve the following optimization problem

$$\begin{aligned} \min _{F,h} E(F, h) = \min _{F, h}\{ \Vert h \otimes F - I \Vert _{2}^{2} + \alpha \rho (\nabla F) + \beta \Vert h \Vert _{2}^{2} + \gamma \Vert D_{F}\Vert _{0} \}, \end{aligned}$$
(2)

where \(\nabla F = (F_{x}, F_{y})\) denotes all the horizontal and vertical first-order derivatives, \(\Vert \cdot \Vert _{0}\) is the \(\ell _{0}\)-norm of a matrix, \(\Vert \cdot \Vert _{2}\) is the Frobenius norm of a matrix, \(\rho (z) = |z|^{0.8}\) is a heavy tailed function, and \(D_{F}\) is the dark channel of F. The first term represents the reconstruction error, the second term, also known as the hyper-Laplacian prior, adds a constraint on image gradients which can preserve image details better than the \(\ell _{1}\) and \(\ell _{0}\) constraints [9, 10] (the hyper-Laplacian prior models the microscopy image gradient distribution better than \(\ell _{1}\) and \(\ell _{0}\)), the third term gives a regularization constraint on the blur kernel, and the last term denotes the sparse prior of the dark channel of the latent image [6]. \(\alpha \), \(\beta \), and \(\gamma \) are weight parameters.

The dark channel of image F is defined by min-filtering:

$$\begin{aligned} D_{F}(x) = \min _{c}( \min _{ y \in \mathcal {N}(x)} F^{c}(y) ), \end{aligned}$$
(3)

where x and y represent pixel locations, \(\mathcal {N}(x)\) is a local patch centered at x, and \(F^{c}\) denotes the color channel of F. The reason to add the dark channel of the latent focused image as a sparse prior in our optimization function is that focused images (Fig. 2(a)) have more dark pixels (i.e., the pixels with the lowest pixel value) than blurred images (Fig. 2(c)).

Fig. 2.
figure 2

(a, b) Focused image and its dark channel; (c, d) blurred image and its dark channel. Dark channels are computed by a 25 \(\times \) 25 sliding window.

However, combining the hyper-Laplacian prior of image gradients and sparse prior of dark channel together makes it very difficult to solve Eq. 2. Thus, we relax the hyper-Laplacian prior \(\rho (\nabla F)\) by \(\Vert \nabla F\Vert _{0}\) and get the new formulation

$$\begin{aligned} \min _{F,h} E(F, h) = \min _{F, h}\{ \Vert h \otimes F - I \Vert _{2}^{2} + \alpha \Vert \nabla F\Vert _{0} + \beta \Vert h \Vert _{2}^{2} + \gamma \Vert D_{F}\Vert _{0} \}. \end{aligned}$$
(4)

We aim to estimate h based on Eq. 4 and then refocus F using the hyper-Laplacian prior later in Eq. 7. To estimate h from Eq. 4, we alternatively solve the latent deblurred image F:

$$\begin{aligned} \min _{F} E(F) = \min _{F} \{\Vert h \otimes F - I \Vert _{2}^{2} + \alpha \Vert \nabla F\Vert _{0} + \gamma \Vert D_{F}\Vert _{0}\}, \end{aligned}$$
(5)

and the blur kernel h (please check the appendix for solutions on Eqs. 5 and 6):

$$\begin{aligned} \min _{h} E(h) = \min _{h}\{\Vert h \otimes F - I \Vert _{2}^{2} + \beta \Vert h \Vert _{2}^{2}\}. \end{aligned}$$
(6)
Fig. 3.
figure 3

Estimating h. (a) Focused phase contrast image (ground truth). (b) Blur kernel and the defocused image. (c) Estimated blur kernel and refocused image by Eq. 4.

In order to test the effectiveness of this method to estimate h, we blur a phase contrast image from well-aligned optics (Fig. 3(a)) with a known kernel and get the defocused image I (Fig. 3(b)). Using image I, we estimate the blur kernel and the latent focused image using Eq. 4, and the results are shown in Fig. 3(c), from which we can observe that the estimated h is closed to the ground truth, but the F by Eq. 4 has many defects.

We also perform the ablation study to test the importance of each prior term in Eqs. 4 and 5 by deleting either the Sparse Prior of image Gradients (without SPG) or the Sparse Prior of Dark Channel (without SPDC) from the optimization problem. The results are summarized in Fig. 4, from which we can see without the sparse prior of image gradient, the refocused image F is less smooth (Fig. 4(b2)). Without the sparse prior of dark channel, the image F is less focused (Fig. 4(c2)). The latent refocused image can be estimated from Eq. 5 directly, however, as shown in Fig. 4(d2), this method is not effective enough to preserve fine cell details in the latent image.

Fig. 4.
figure 4

Refocusing F. (a) Defocused phase contrast image. (b) Result without SPG in Eq. 5. (c) Result without SPDC in Eq. 5. (d) Result by Eq. 5. (e) Result by Eq. 7.

Refocusing F: With the estimated h from Eq. 4, we formulate a non-blind refocusing problem with the hyper-Laplacian prior,

$$\begin{aligned} \min _{F}\{\Vert h \otimes F - I \Vert _{2}^{2} + \delta \rho (\nabla F)\}, \end{aligned}$$
(7)

where \(\delta \) is a weight parameter.

The hyper-Laplacian prior makes Eq. 7 a non-convex optimization problem, which is commonly regarded as computationally intractable. To solve this problem, an iterative reweighted least square process [9], which poses the optimization as a sequence of least square problems while the weight of each derivative is updated based on the previous iteration, is implemented. As shown in Fig. 4(e2), the hyper-Laplacian prior has better power to preserve specimen details than \(\Vert \nabla F\Vert _{0}\) (Fig. 4(d2)).

2.2 Refocusing Images from the Specimen Perspective

Problem Formulation: From Fig. 4(e), we can see that after getting F with the hyper-Laplacian prior, there are still some small wavy artifacts in the background (see Fig. 5(d)). As the refocused F can be regarded as an image produced from the properly-aligned optics, we can build a linear imaging model as

$$\begin{aligned} F = h_{opt} \otimes (L + S) + n, \end{aligned}$$
(8)

where \(h_{opt}\) is from [7] (a mathematically-derived PSF based on well-aligned optics), L is the artifact-free image, and S is the artifact image.

Solving for L and S: Considering the sparse property of the artifacts and the smoothness of the artifact-free image, we formulate the following optimization problem to solve L and S:

$$\begin{aligned} \min _{L,S} E(L,S) = \min _{L,S} \{\Vert h_{opt} \otimes (L+S) - F \Vert _{2}^{2} + \lambda \Vert \nabla L\Vert _{0} + \mu \Vert S\Vert _{0}\}, \end{aligned}$$
(9)

where \(\lambda \) and \(\mu \) are weight parameters. By fixing L first, we can alternatively solve the artifact image S:

$$\begin{aligned} \min _{S} E(S) = \min _{S} \{\Vert h_{opt} \otimes L + h_{opt} \otimes S - F \Vert _{2}^{2} + \mu \Vert S\Vert _{0}\}, \end{aligned}$$
(10)

and then update the artifact-free image L:

$$\begin{aligned} \min _{L} E(L) = \min _{L} \{\Vert h_{opt} \otimes L + h_{opt} \otimes S - F \Vert _{2}^{2} + \lambda \Vert \nabla L\Vert _{0}\}. \end{aligned}$$
(11)

The iterative solutions of Eqs. 10 and 11 can be found in the appendix.

Figure 5 shows that after refocusing the image on optics, the edges and nuclei become much sharper (Fig. 5(b)), but there are wavy artifacts in the background. After further refocusing the image on specimens by assuming artifacts are sparse and artifact-free image is smooth, the artifacts in the background (Fig. 5(d)) are removed and the specimens are presented more clearly (Fig. 5(c)), which also proves the efficiency of our assumption.

Fig. 5.
figure 5

Refocusing from the perspectives of optics and specimens. (a) Input image I. (b) F obtained by refocusing the optics only. (c) L obtained by refocusing the optics and specimens. (d) Removed artifacts S.

3 Experimental Results

Dataset: 500 phase contrast microscopy images with different cell densities were captured at the resolution of \(1040 *1392\) pixels.

Evaluation Metrics: To evaluate our refocusing-optics step, we use the Structural Similarity Index (SSIM) [12]. To evaluate our refocusing-specimens step, we test how well it can facilitate the cell segmentation task using the accuracy metric. By denoting cell and background pixels as positive (P) and negative (N), respectively, the accuracy is defined as \(ACC = (|TP| + |TN|)/(|TP| + |FP| + |TN| + |FN|)\), where TP is the true positive, FP is the false positive, TN is the true negative, and FN is the false negative.

Parameter Setup: In our experiments, 100 images are used to learn the parameters by 5-fold cross validation. The rest of the dataset is used for testing. The three parameters \(\alpha \), \(\beta \), and \(\gamma \) in Eq. 4 (estimating blur kernel h) are \(4e^{-3}\), \(1e^{-4}\), and \(4e^{-3}\), respectively. The parameter \(\delta \) in Eq. 7 (estimating optics-refocused F) is \(5e^{-4}\). The parameters \(\lambda \) and \(\mu \) in Eq. 9 (estimating L with refocused optics and specimens) are \(3e^{-4}\) and \(5e^{-4}\), respectively.

Fig. 6.
figure 6

Qualitative evaluation.

Evaluation: Figure 4 qualitatively compares our refocusing-optics algorithm with alternative methods. We quantitatively compare them using the SSIM index. As shown in Table 1, our method (Eq. 7) that estimates h first and then estimates the optics-refocused F, outperforms the other methods.

Figures 5 and 6(a1, b1, c1) show the superior performance of our algorithm on refocusing images, which provides a sharp visibility on specimens. To demonstrate how our refocusing algorithm facilitates automated microscopy image analysis, we use the cell segmentation as a case study. Based on our refocused image (Fig. 6(c1)), the simple Otsu thresholding method [13] is used to segment specimens from the background. As shown in Fig. 6(c2), specimens can be easily segmented from the background. Furthermore, Fig. 6(c2) (segmentation of the optics-refocused and specimen-refocused image, Fig. 6(c1)) has less noise than Fig. 6(b2) (segmentation of the optics-refocused image, Fig. 6(b1)).

We quantitatively compare our results (e.g., Fig. 6(c2)) with [7] (e.g., Fig. 6(d)) and [8] (e.g., Fig. 6(e)) using the accuracy index. The ground truth segmentation (Fig. 6(a3)) is obtained by thresholding the corresponding fluorescence image (Fig. 6(a2)) (note that, in real experiments we do not use chemical stains to damage cells’ viability. The fluorescence images are only used for our groundtruth purpose). As shown in Table 2, the segmentation results using our method over 400 testing images are more accurate because our refocused images focus image contents on specimens with the uniform background whose artifacts are removed.

Table 1. Quantitative evaluation of the refocusing optics methods.
Table 2. Quantitative evaluation of the segmentation results.

4 Conclusion

In this paper, we investigate a refocusing algorithm to refocus the phase contrast image from two perspectives. First, given a defocused phase contrast microscopy image caused by misaligned optics, we estimate the blur kernel by implementing a blind deblurring algorithm with the dark channel sparse prior, and then unblindly refocus the image with the hyper-Laplacian prior of image gradient. Secondly, we remove artifacts from the optics-refocused image to enhance the contrast between specimens and background using the intrinsic point spread function of the phase contrast microscopy image and the sparse prior of artifacts. Note that, if the input defocused image is indeed well-focused, our refocusing-optics step will return a Dirac delta function for h and the optics-refocused image will be identical to the input.

The preliminary experiments demonstrate that our algorithm is very effective to refocus phase contrast microscopy images. After refocusing the image from both the optics and specimen perspectives, the refocused image provides better visualization on specimen details and facilitates automated cell image analysis.