Consensus-based clustering for document image segmentation

Abstract

Segmentation of a document image plays an important role in automatic document processing. In this paper, we propose a consensus-based clustering approach for document image segmentation. In this method, the foreground regions of a document image are grouped into a set of primitive blocks, and a set of features is extracted from them. Similarities among the blocks are computed on each feature using a hypothesis test-based similarity measure. Based on the consensus of these similarities, clustering is performed on the primitive blocks. This clustering approach is used iteratively with a classifier to label each primitive block. Experimental results show the effectiveness of the proposed method. It is further shown in the experimental results that the dependency of classification performance on the training data is significantly reduced.

Appendices

Appendix

Extension of the consensus-based clustering approach for multiple number of features

The proposed segmentation method is developed on the feature set $\mathbf {Z}=\{\mathcal {C},\mathcal {S}\}$. However, this method can be extended for n number of features, with some modifications in the consensus-based clustering approach. Equation (4) can be replaced with Eq. (7), where $w_{z}$ is the weight associated with the feature $z \in \mathbf {Z}$.

$$\begin{aligned} sim_{ij} = \frac{\sum _{z\in \mathbf {Z}}{w_{z}S_{ij}^{z}}}{\sum _{z\in \mathbf {Z}}{w_{z}}} \end{aligned}$$

(7)

$sim_{ij}$ in Line 5 of Algorithm 2 is computed using Eq. (7). In Algorithm 4, Lines $1-4$ can be replaced with a for loop iterating for each $z \in \mathbf {Z}$ to compute the graph $G_{z}(V_z,E_z)$ using the set of weights W. In each iteration, W contains weights $w_z=1$ and $\forall z' \in \{\mathbf {Z}{\setminus }\{z\}\}$, $w_{z'}=0$. Similar modification is required in Algorithm 3 for Lines 2–7. In this for loop, set of connected nodes $\mathscr {N}_{z}$ for each $G_{z}$ is also computed. Collection of all such $\mathscr {N}_{z}$ is stored in $\mathbf {N}$. The for loop of Lines 8–15 in this algorithm is replaced with the function call FindClusters($K_{CCCN},A,\mathbf {N},z,t$), where z and t are initialized with 1. The function FindClusters is given in Algorithm 7.

This algorithm takes $K,D,\mathbf {N},z,$ and t as inputs. K is the set of clusters which is initialized as an empty set and gets updated in each recursive call of this function. $\mathbf {N}$ is the collection of sets of connected nodes $\mathscr {N}_{z}$ obtained from the graph $G_z(V_z,E_z)$ for each $z \in \mathbf {Z}$. The set D is generated by taking a single element from each $\mathscr {N}_{z}$ at a time. Therefore, |D| = $|\mathbf {N}|$. The variables z and t denote indices for elements of $\mathbf {N}$ and D, respectively. In this algorithm, Lines 1–7 populate D recursively with $|\mathbf {N}| - 1$ elements, with only one element from each $\mathscr {N}_{z}$. Recursive call of this function is made in Line 5. This recursive call ensures filling $\mid \mathbf {N} \mid - 1$ positions of D with $\prod _{z=1}^{\mid \mathbf {N} \mid - 1}$ number of combinations. Lines 8–16 represent the base condition of the recursive function. The for loop of Lines 9–15 fills the last position of D by selecting one by one element from the last member of $\mathbf {N}$ and computes intersection among the elements of D. If the intersection results in a non-empty set, then the set of clusters K is updated with the non-empty set (Lines 12–14).

Clustering metrics

For evaluating a clustering algorithm, adjusted rand index (ARI) was used in [29]. It is computed according to Eq. (8). For computing ARI, let the ground truth classes be represented as, $C=\{c_1, c_2, \ldots , c_{|C|}\}$. Computed clusters are represented as, $K=\{k_1, k_2, \ldots ,k_{|K|}\}$. However, the ground truth classes and the computed clusters are represented in such a way that $\sum _{i=1}^{|C|} |c_i|$ $=$ $\sum _{j=1}^{|K|} |k_j|$. Let the total number of blocks present in an input document image that needs to be clustered, be n, where n $=$ $\sum _{i=1}^{|C|} |c_i|$. There are $n \atopwithdelims ()2$ number of possible combinations for n number of blocks. These $n \atopwithdelims ()2$ combinations can be subdivided into four classes: true positive (tp), false positive (fp), true negative (tn), and false negative (fn), such that $n \atopwithdelims ()2$ $=$ $tp + fp + tn + fn$. A pair is considered to be in tp, if both blocks in the pair are present in the same class in C and in the same cluster in K. If the blocks in the pair are present in different classes in C and in the same cluster in K, then the pair is said to be fp. If blocks in the pair are present both in different classes in C and in different clusters in K, then the pair is considered to be tn. A pair belongs to fn, if they are in different clusters in K, but in the same class in C.

$$\begin{aligned}&\mathrm{ARI} = \frac{\mathrm{Index} - \mathrm{ExpectedIndex}}{\mathrm{MaximumIndex} - \mathrm{ExpectedIndex}} \end{aligned}$$

(8)

$$\begin{aligned}&\text {where,} \quad \mathrm{Index} = \sum _{i = 1}^{|C|} \sum _{j = 1}^{|K|} {|x_{ij}| \atopwithdelims ()2}, \nonumber \\&x_{ij} = c_i \cap k_j, \quad c_i \in C, \quad \text {and} \quad k_j \in K \end{aligned}$$

(9)

$$\begin{aligned}&\mathrm{ExpectedIndex} = \frac{(tp + fp) \times (tp + fn) }{tp + fp + tn + fn} \end{aligned}$$

(10)

$$\begin{aligned}&\mathrm{MaximumIndex} = \frac{(tp + fp) + (tp + fn)}{2} \end{aligned}$$

(11)

To evaluate a clustering algorithm, an entropy-based technique is proposed in [45], known as V-measure (V). V-measure is the harmonic mean of homogeneity ($\varsigma$) and completeness ($\varphi$) (defined later). It is given in Eq. (12), where $\beta$ is positive.

$$\begin{aligned} V_{\beta } = \frac{(1 + \beta ) \times \varsigma \times \varphi }{(\beta \times \varsigma ) + \varphi } \end{aligned}$$

(12)

To achieve high $\varsigma$ measure, a clustering algorithm needs to cluster datapoints from the same class to the same cluster. It is computed using Eq. (13). To achieve high $\varphi$ measure, the clustering algorithm must cluster all datapoints from the same class to the same cluster. $\varphi$ measure is expressed using Eq. (14). In these expressions, H(C|K) and H(K|C) quantify relative entropy between two variables, and H(C) and H(K) represent entropy of a variable. Relative entropy between two variables C and K is expressed in Eq. (15), and entropy of a variable C is defined in Eq. (16).

$$\begin{aligned}&\varsigma = \left\{ \begin{array}{l l} 1 &{} \quad \text {if }H(C,K) = 0\\ 1 - \frac{H(C|K)}{H(C)} &{} \quad \text {otherwise}\\ \end{array} \right. \end{aligned}$$

(13)

$$\begin{aligned}&\varphi = \left\{ \begin{array}{l l} 1 &{} \quad \text {if }H(K,C) = 0\\ 1 - \frac{H(K|C)}{H(K)} &{} \quad \text {otherwise}\\ \end{array} \right. \end{aligned}$$

(14)

$$\begin{aligned}&H(C|K) = - \sum _{i = 1}^{|K|}{\sum _{j = 1}^{|C|}{\frac{|x_{ij}|}{n}}\mathrm{log}\left( \frac{|x_{ij}|}{\sum _{i=1}^{|C|}{|x_{ij}|}}\right) }\end{aligned}$$

(15)

$$\begin{aligned}&H(C) = - \sum _{i = 1}^{|C|}{\frac{\sum _{j = 1}^{|K|}{|x_{ij}|}}{n}\mathrm{log}\left( \frac{\sum _{j = 1}^{|K|}{|x_{ij}|}}{n}\right) } \end{aligned}$$

(16)

Adjusted mutual information (AMI) has been proposed in [50] and is defined using Eq. 17.

$$\begin{aligned}&\mathrm{AMI} = \frac{\mathrm{MI}(C,K) - E(\mathrm{MI}(C,K))}{\mathrm{max}(H(C),H(K)) - E(\mathrm{MI}(C,K))} \end{aligned}$$

(17)

$$\begin{aligned}&\mathrm{MI}(C,K) = \sum _{i = 1}^{|C|}\sum _{j = 1}^{|K|} \frac{|x_{ij}|}{n}log \left( \frac{\frac{|x_{ij}|}{n}}{\sum _{l = 1}^{|K|}\frac{|x_{il}|}{n}\sum _{l = 1}^{|C|}\frac{|x_{lj}|}{n}}\right) \end{aligned}$$

(18)

$$\begin{aligned}&E(\mathrm{MI}(C,K) = \sum _{i = 1}^{|C|}\sum _{j = 1}^{|K|}\sum _{|x_{ij}|}\frac{|x_{ij}|}{n} log\left( \frac{n|x_{ij}|}{|c_i||k_j|}\right) P \nonumber \\&\quad \text {where,} \quad P = \frac{{n \atopwithdelims ()|x_{ij}|}{n-|x_{ij}| \atopwithdelims ()|c_i|-|x_{ij}|}{n-|c_i| \atopwithdelims ()|k_j|-|x_{ij}|}}{{n \atopwithdelims ()|c_i|}{n \atopwithdelims ()|k_j|}} \end{aligned}$$

(19)

Multi-class classification metrics

For evaluation of multi-class classification, four metrics have been used, namely average accuracy ($\mathrm{Avg}_{\mathrm{Acc}}$ Eq. 20), error rate ($\mathrm{Err}_{\mathrm{Rate}}$ Eq. 21), $\mathrm{FScore}_{\mu }$ (Eq. 22), and $\mathrm{FScore}_{M}$ (Eq. 23). All these equations are adopted from [46].

$$\begin{aligned}&\mathrm{Avg}_{\mathrm{Acc}} = \frac{\sum _{i=1}^{\mid \mathfrak {I}\mid }{\frac{tp_i+tn_i}{tp_i+fp_i+tn_i+fn_i}}}{\mid \mathfrak {I}\mid } \end{aligned}$$

(20)

$$\begin{aligned}&\mathrm{Err}_{\mathrm{Rate}} = \frac{\sum _{i=1}^{\mid \mathfrak {I}\mid }{\frac{fp_i+fn_i}{tp_i+fp_i+tn_i+fn_i}}}{\mid \mathfrak {I}\mid }\nonumber \\&\mathrm{FScore}_{\mu } = \frac{(\beta ^{2}+1)\mathrm{Precision}_{\mu }\mathrm{Recall}_{\mu }}{\beta ^{2}\mathrm{Precision}_{\mu }+\mathrm{Recall}_{\mu }} \nonumber \\&\text {where,} \quad \mathrm{Precision}_{\mu } = \frac{\sum _{i=1}^{\mid \mathfrak {I}\mid }{tp_i}}{\sum _{i=1}^{\mid \mathfrak {I}\mid }{tp_i+fp_i}} \end{aligned}$$

(21)

$$\begin{aligned}&\text {and} \quad \mathrm{Recall}_{\mu } = \frac{\sum _{i=1}^{\mid \mathfrak {I}\mid }{tp_i}}{\sum _{i=1}^{\mid \mathfrak {I}\mid }{tp_i+fn_i}} \nonumber \\&\mathrm{FScore}_{M} = \frac{(\beta ^{2}+1) \mathrm{Precision}_{M}\mathrm{Recall}_{M}}{\beta ^{2}\mathrm{Precision}_{M}+\,\mathrm{Recall}_{M}} \nonumber \\&\text {where,} \quad \mathrm{Precision}_{M} = \frac{\sum _{i=1}^{\mid \mathfrak {I}\mid }{\frac{tp_i}{tp_i+fp_i}}}{\mid \mathfrak {I}\mid } \end{aligned}$$

(22)

$$\begin{aligned}&\text {and} \quad \mathrm{Recall}_{M} = \frac{\sum _{i=1}^{\mid \mathfrak {I}\mid }{\frac{tp_i}{tp_i+fn_i}}}{\mid \mathfrak {I}\mid } \end{aligned}$$

(23)

In Eqs. (20–23), $|\mathfrak {I}|$ represents the total number of classes, and $tp_i$, $fp_i$, $tn_i$, and $fn_i$, respectively, represent, true-positive rate, false-positive rate, true-negative rate, and false-negative rate for the $i{\mathrm{th}}$ class. $\mu$ and M represent micro- and macro- averaging. During evaluation, micro-averaging gives preference to bigger classes, whereas macro-averaging treats all classes equally.