Adaptive total variation based image segmentation with semi-proximal alternating minimization

Highlights

• We propose a novel adaptive total variation-based segmentation model combining the weighted matrix with the gradient operator.

• Our model has a unique minimizer and can be solved effectively by the semi-proximal alternating direction method of multipliers with guaranteed convergence.

• We utilize the K-means method to select the thresholds automatically and get good restoration and segmentation results for complex situations.

Abstract

To improve the image segmentation quality, it is important to adequately describe the local features of targets in images. In this paper, we develop a novel adaptive total variation based two-stage segmentation approach to restore and segment images under complex degradations. To find a smooth approximation solution in the first stage, we introduce an effective regularization term that combines an adaptive weighted matrix with the gradient operator. The adaptive weighted matrix gives different penalties in the axis directions to enhance the diffusion along the tangent direction of the edge. It can filter out the details far away from the edge and preserve the main structure of targets. For the convex objective function in the first stage, a semi-proximal alternating direction method of multipliers (sPADMM) with guaranteed convergence is successfully employed. We utilize the K-means method to select thresholds automatically and complete the segmentation by thresholding the image into different regions in the second stage. Extensive experimental comparisons between our method and some state-of-the-art methods including a deep learning approach are provided. All numerical results illustrate clearly that our method has better performance for different kinds of segmentation and restoration tasks.

Introduction

Image segmentation has always been a hot topic, which aims to divide an image domain into mutually disjoint regions reasonably. This plays a significant role in many application fields, such as medical imaging [1], [2], vehicle license plate recognition [3], [4], etc. During the past decades, many excellent approaches have been proposed [5], [6], [7], [8], [9], [10]. In 1989, one of the most important segmentation models called MS model was proposed by Mumford and Shah [6], which was a landmark achievement in segmentation fields. Let $\Omega \subset {\mathbb{R}}^2$ be a bounded open connected set, and $\Gamma$ be a compact curve in $\Omega ,$ the MS model can be formulated as

$$E_{\mathrm{MS}}(u,\Gamma ;\Omega )=\frac{\lambda }{2}{\int }_{\Omega }{(f-u)}^{2}dx+\frac{\mu }{2}{\int }_{\Omega \setminus \Gamma }{\left|\nabla u\right|}^{2}dx+\text{Length}\left(\Gamma \right),$$

where $\lambda ,\mu$ are positive parameters, $f:\Omega \to \mathbb{R}$ is the degraded image and $u:\Omega \to \mathbb{R}$ is the ideal image. Here, the length of $\Gamma$ can be written as ${\mathcal{H}}^1\left(\Gamma \right),$ i.e., the 1-dimensional Hausdorff measure in ${\mathbb{R}}^2$. Model (1) adopted the energy minimization equation to find an approximate solution, but it is difficult to solve due to its nonconvexity.

Some early attempts [11], [12] to solve (1) were done by approximating it using a sequence of simpler elliptic problems. Meanwhile, more works have focused on exploring the MS model such as active contour methods [13], [14], graph cut methods [15] and convex relaxation approaches [16], [17], etc. In 2013, Cai et al. proposed a two-stage image segmentation strategy using a convex variant of the MS model and thresholding [18], which got good segmentation results. The first stage is to find a smooth approximation solution $u$ by minimizing the energy functional:

$$\underset{u}{inf}\{\frac{\lambda }{2}{\int }_{\Omega }{(f-\mathcal{A}u)}^{2}dx+\frac{\mu }{2}{\int }_{\Omega }{\left|\nabla u\right|}^{2}dx+{\int }_{\Omega }\left|\nabla u\right|dx\},$$

where $\mathcal{A}$ is the problem related linear operator. Once $u$ is obtained, the segmentation is done by thresholding $u$ properly in the second stage. Since then, many novel image segmentation methods [19], [20], [21], [22], [23], [24] have been proposed based on the two-stage idea [18], and they show the subtle connection between image restoration and segmentation to some extent. The two-stage strategy and corresponding improved models have noticeable virtues in image segmentation, refer to [18], [19], [23] for details.

As we know, image restoration (including denoising, deblurring, inpainting, etc.) is to estimate a clean image from the degraded image. And the segmentation can be used as the preprocessing or postprocessing for restoration. In [19], Chan et al. used the two-stage method for segmenting blurry images with Poisson or multiplicative Gamma noise. Duan et al. [20] introduced Euler’s Elastica as the regularization in the MS model based on a two-stage segmentation strategy to capture the geometry of object shapes in noisy images with missing pixels. In [24], the discrete-MS model was proposed, and the proximal alternating linearized minimization algorithm was employed to deal with the objective function to restore a degraded image and extract its contours. In [23], authors explored a linkage between the PCMS model [6] and the ROF model [25], then derived a novel thresholded-ROF (T-ROF) segmentation method to illustrate the advantages of image segmentation through image restoration techniques. However, as the regularizers of the aforementioned methods are isotropic, non-ideal results can be generated, especially when images with complex geometry structures.

Many previous works [18], [25], [26], [27] have considered the total variation (TV) regularization and anisotropic filtering as standard methods for image restoration because of their capability to detect and maintain image edges. However, the anisotropic filtering produces artifacts and the TV regularization causes the staircase effect [25], [28]. In order to overcome these drawbacks, many approaches [28], [29], [30], [31], [32] were proposed by combining the TV regularization and anisotropic filtering, where the isotropic TV norm is replaced by an anisotropic term. For example, Pang et al. proposed an anisotropic TV-based restored model combining a weighted matrix with the gradient operator [30]. A model based on the adaptive weighted TV${}^p$ regularization in [31] was subsequently proposed for image denoising, where the rotation matrix and weight matrix are embedded in TV${}^p$ regularization.

Different from the reweighted $l_1$ algorithm [33], [34], the weighted $l_1-l_2$ algorithm [35], [36], TV${}^p$ [37], [38], [39] and the nuclear/$l_{2,1}$-norm regularization [40], we often need that the target model diffuses along the tangent direction of the local features. Generally, the setting for ${\nabla }_x$ and ${\nabla }_y$ with the same weight cannot efficiently couple with the local features. To manifest the geometry structures of targets in images adequately, we give different penalties in the $x$-axis and $y$-axis directions according to the gradient information ${\nabla }_x$ and ${\nabla }_y$. Based on the above facts, we propose an adaptive TV-based two-stage segmentation model with the following novelties:

• We present a robust and efficient anisotropic segmentation model, where the regularization combines the adaptive weighted matrix $T$ (see details in Section 2.2) and gradient operator (${\nabla }_x,$ ${\nabla }_y$) to describe geometry structure details and more edge information of segmentation objects in images. The adaptive weighted matrix $T$ has good properties in capturing elongated fine structures, which are useful for fine structure segmentation.

• To deal with the adaptive TV-based convex objective function, an elegant and effective numerical method is designed through the sPADMM algorithm with guaranteed convergence [41], [42]. Furthermore, extensive experiments demonstrate that the proposed algorithm and model outperform advanced methods on kinds of image segmentation problems under various degraded situations such as noise, blur and missing pixels, etc.

The outline of this paper is as follows. In Section 2, we introduce our anisotropic segmentation model and present the properties of the model. In Section 3, the sPADMM algorithm is carefully applied to solve the convex optimization model and the convergence analysis of the algorithm is shown. We list some preparation works of numerical experiments in Section 4. In Section 5, numerical examples are presented to compare different methods including a celebrated U-Net approach. Finally, the conclusions are drawn in Section 6.