Conservative preprocessing of document images

Abstract

Many preprocessing techniques intended to normalize artifacts and clean noise induce anomalies in part due to the discretized nature of the document image and in part due to inherent ambiguity in the input image relative to the desired transformation. The potentially deleterious effects of common preprocessing methods are illustrated through a series of dramatic albeit contrived examples and then shown to affect real applications of ongoing interest to the community through three writer identification experiments conducted on Arabic handwriting. Retaining ruling lines detected by multi-line linear regression instead of repairing strokes broken by deleting ruling lines reduced the error rate by 4.5 %. Exploiting word position relative to detected rulings instead of ignoring it decreased errors by 5.5 %. Counteracting page skew by rotating extracted contours during feature extraction instead of rectifying the page image reduced the error by 1.4  %. All of these accuracy gains are shown to be statistically significant. Analogous methods are advocated for other document processing tasks as topics for future research.

Appendix

For binary classification errors [18], we define:

• Type I (false positive): detecting a class that is not present.

• Type II (false negative): failing to detect a class that is present.

One often needs to compare the accuracy of one classification algorithm with that of another. According to Dietterich’s study of five statistical significance tests, McNemar’s [14] has a low probability of incorrectly detecting a difference when no difference exists.

Suppose there are two algorithms, baseline $\mathcal {A}$ and proposed $\mathcal {B}$. The available n samples are classified by both algorithms. It is observed that $n_{10}$ of the samples are misclassified by classifier $\mathcal {A}$ but not by $\mathcal {B}$, $n_{01}$ samples are misclassified only by $\mathcal {B}$, and $n_{11}$ samples are misclassified by both algorithms. The accuracy of $\mathcal {A}$ is $\mathcal {A} = (n - n_{10} - n_{11}) / n$, and the accuracy of $\mathcal {B}$ is $\mathcal {B} = (n - n_{01} - n_{11}) / n$. McNemar’s test is formulated as:

$$\begin{aligned} Z^{2} = \frac{(| n_{10} - n_{01} | - 1)^{2}}{n_{10} + n_{01}}. \end{aligned}$$

(5)

where $n_{01}$, number of samples misclassified by the proposed algorithm $\mathcal {B}$, but not by the baseline $\mathcal {A}$; $n_{10}$, number of samples misclassified by the baseline $\mathcal {A}$, but not by the proposed algorithm $\mathcal {B}$; null hypothesis $\mathcal {H}_{0}$, $\mathcal {A}$ $=$ $\mathcal {B}$; alternative hypothesis $\mathcal {H}_{1}$, $\mathcal {A} < \mathcal {B}$.

The test statistic $Z^{2}$ approximately follows the Chi-square distribution with one degree of freedom. As a rule of thumb, we say one algorithm outperforms another significantly with a confidence level of 95 %. The test value $Z^{2}$ corresponding to this 95 % confidence is 3.84. Although this statistic test is approximate, it is effective in detecting accuracy differences between algorithms [14].

As an example of the application of McNemar’s hypothesis test, consider two cases:

• Case 1: $n_{01} = 10$, $n_{10} = 20$, therefore $Z^{2} = 2.70$. We cannot conclude that $\mathcal {B}$ is significantly better than $\mathcal {A}$.

• Case 2: $n_{01} = 5$, $n_{10} = 15$, $Z^{2} = 4.05$. Therefore, $\mathcal {B}$ is more accurate than $\mathcal {A}$ at a confidence level of 95 %.

Note that if $n_{11} = 20$ and $n=100$ in both cases, then in Case 1 $\mathcal {A} = 60\,\%$ and $\mathcal {B} = 70\,\%$. In Case 2, $\mathcal {A}=65\,\%$ and $\mathcal {B} = 75\,\%$. Different values of $n_{11}$, the number of samples misclassified by both algorithms, would give different accuracies for $\mathcal {A}$ and $\mathcal {B}$, but that would not change our conclusion with respect to their comparative accuracy. At low error rates and large sample sizes, small differences in accuracy can be statistically significant. It is, of course, essential to keep track of the specific errors. The commonly used recall and precision measure does not provide sufficient information for this test.

All of the differences in error rate reported as significant in Sects. 4 and 5 yielded confidence greater than 95 % with McNemar’s test.