BOB: a bi-level overlapped binning procedure for scene word binarization

Abstract

Scene text analysis involves detecting and processing text/words in natural scene images for serving various purposes. This problem domain intrigues the research fraternity due to challenges like dealing with noise, blur, heterogeneous intensity variation, etc. The ultimate goal is making detected scene word recognizable by any standard Optical Character Recognition system, thereby necessitating effective scene word binarization. Several methods address scene text detection, but comparatively few addresses scene word binarization. These binarization methods, however, have limitations in robustness against image quality-based complexities thus causing low precision. Here, a novel approach is proposed for scene word binarization called Bi-level Overlapped Binning where intensities of color channels R, G and B are grouped or binned to generate several solutions in the form of binary images. The stable binary images are identified such that the image solutions from them can be classified as text or non-text using a standard classifier trained with some popular features. Finally, the resultant text solutions are combined probabilistically to get the binarized output. The proposed method is evaluated on standard datasets such as SVT, ICDAR-2003, ICDAR-2011 (Scene), ICDAR-2011 (BDI), KAIST and Total-Text achieving precisions 0.76, 0.87, 0.89, 0.85, 0.84 and 0.87 respectively, which are mostly better than that of the state-of-the-art.

Appendix

Lemma 1: If the difference between the pixel with the highest PixelValue and lowest PixelValue in foreground region R is D and let S be any bin size such that S ≥ D then for all (i, j) ∈ R, (${\varDelta}_k^s$)_ij = 1 for some bin number $k\in \left[1,2\ast m-1\right], where\ m=\left\lfloor \frac{PV_{max}}{S}\right\rfloor +1$.

Given Definitions:

• 1. Definition of a bin image ${B}_k^s$ where ${\left({B}_k^s\right)}_{ij}$ is the value at coordinate (i, j) of the image:

$$ {\left({\mathrm{B}}_k^s\right)}_{ij}\stackrel{\scriptscriptstyle\mathrm{def}}{=}\left\{\begin{array}{c}1,k\in \left[1,m\right]\kern0.50em and\ \left\lfloor \frac{PixelValue\left(i,j\right)}{S}\right\rfloor +1=k\ \\ {}1,k\in \left[m+1,2\times m-1\right]\ and\ \left\lfloor \frac{PixelValue\left(i,j\right)-\frac{S}{2}}{S}\right\rfloor +1=k-m\\ {}0, otherwise\end{array}\right. $$

(12)

• 2. Definition of a Delta image ${\varDelta}_i^s$ :

$$ {\displaystyle \begin{array}{cc}\mathbf{Level}\ \mathbf{1}\ \boldsymbol{\Delta}\ \mathbf{bins}:& \kern3.25em {\Delta}_i^s={\mathrm{B}}_i^s\cup {\mathrm{B}}_{i+m}^s,\kern0.5em \mathrm{for}\ 1\le i<m\\ {}\mathbf{Level}\ \mathbf{2}\ \boldsymbol{\Delta}\ \mathbf{bins}:& \kern3.25em {\Delta}_i^s={\mathrm{B}}_i^s\cup {\mathrm{B}}_{i-m+1}^s,\mathrm{for}\ m<i<2\times m\end{array}} $$

(13)

• 3. Definition of a foreground region R:

A foreground region R is a set of two or more points in an image I such that for every point p in R, there exists a point q in R such that q is in N₈(p) where N₈ is the set of 8-connected neighbors of a point.

Proof: Let the minimum PixelValue for the region R be P_min for pixel coordinate (x_a, y_a) and maximum be P_max for pixel coordinate (x_b, y_b) such that P_max − P_min = D.

Let,

$$ k\stackrel{\scriptscriptstyle\mathrm{def}}{=}\left\lfloor \frac{P_{min}}{S}\right\rfloor +1 $$

(14)

From the definition of (${B}_k^s$)_ij, in Eq (12) ${\left({B}_k^s\right)}_{XaYa}=1$. All pixels with PixelValue equal to P_min in R will occur as a positive (value = 1) in binary image ${B}_k^s$

(Now)

$$ {P}_{max}-{P}_{min}=D\Rightarrow {\boldsymbol{P}}_{\boldsymbol{max}}={\boldsymbol{P}}_{\boldsymbol{min}}+\boldsymbol{D} $$

(Thus,)

$$ \left\lfloor \frac{P_{max}}{S}\right\rfloor +1=\left\lfloor \frac{P_{min}+D}{S}\right\rfloor +1=\left\lfloor \frac{P_{min}}{S}+\frac{D}{S}\ \right\rfloor +1\le \left\lfloor \frac{P_{min}}{S}+1\ \right\rfloor +1=\left\lfloor \frac{P_{min}}{S}\right\rfloor +1+1=k+1 $$

(Since,)

$$ D\le S $$

$$ \therefore \kern2.25em \left\lfloor \frac{P_{max}}{S}\right\rfloor +1\le \mathrm{k}+1 $$

(15)

Also,

$$ \left\lfloor \frac{P_{max}}{S}\right\rfloor +1\ge \left\lfloor \frac{P_{min}}{S}\right\rfloor +1=k $$

(16)

Thus, from Eq. (15) and Eq. (16) we get,

$$ {\displaystyle \begin{array}{c}k\le \frac{P_{max}}{S}+1\le k+1\\ {}\Rightarrow \frac{P_{max}}{S}+1=k\kern0.5em \mathrm{OR}\kern0.5em \Rightarrow \frac{P_{max}}{S}+1=k+1\end{array}} $$

(17)

Since k is an integer and $\left\lfloor \frac{P_{max}}{S}\right\rfloor +1$ is an integer.

The pixels in foreground region R with value P_max will fall in the same bin as the pixels with value P_min (bin number k) or the immediate next bin (bin number k + 1). Since difference is less than the size of each bin, the bins where the region R is spread over, are limited to a single bin or two adjacent bins:

$$ \mathbf{Case}\ \mathbf{1}:\left\lfloor \frac{P_{max}}{S}\right\rfloor +1=k. $$

In that case, both $\left\lfloor \frac{P_{min}}{S}\right\rfloor \kern0.5em +1=k\kern0.5em$from Eq. (3) and $\left\lfloor \frac{P_{max}}{S}\right\rfloor +1=k$

For any (i, j) ∈ R,

$$ {P}_{min}\le PixelValue\left(i,j\right)\le {P}_{max} $$

$$ \Rightarrow \kern0.75em \left\lfloor \frac{P_{min}}{S}\right\rfloor +1\le \left\lfloor \frac{PixelValue\left(i,j\right)}{S}\right\rfloor +1\le \kern0.75em \left\lfloor \frac{P_{max}}{S}\right\rfloor +1 $$

$$ \Rightarrow \kern2.5em k\le \left\lfloor \frac{PixelValue\left(i,j\right)}{S}\right\rfloor +1\le k $$

$$ \therefore \left\lfloor \frac{PixelValue\left(i,j\right)}{S}\right\rfloor +1=k $$

Since k is an integer and$\left\lfloor \frac{PixelValue\left(i,j\right)}{S}\right\rfloor +1$ is an integer, this is the only possible solution

$$ \Rightarrow {\left({\mathrm{B}}_k^s\right)}_{\mathrm{ij}}=1\ for\ all\ \left(i,j\right)\in R $$

(Now,)

$$ {\Delta}_k^s={\mathrm{B}}_k^s\cup {\mathrm{B}}_{k+m}^s $$

Since

$$ {\left({\mathrm{B}}_k^s\right)}_{\mathrm{ij}}=1\ \mathrm{for}\ \mathrm{all}\ \left(i,j\right)\in R $$

$$ \therefore {\left({\Delta}_k^s\right)}_{\mathrm{ij}}=1\ \mathrm{for}\ \mathrm{all}\ \left(i,j\right)\in R $$

$$ \mathbf{Case}\ \mathbf{2}:\left\lfloor \frac{P_{max}}{S}\right\rfloor +1=k+1 $$

(18)

We have to consider two different sub cases depending on the value of P_max

$$ \mathbf{Case}\ \mathbf{2.1}:{P}_{max}-\left(k\ast S\right)<\frac{S}{2} $$

(19)

Let us consider bin number k + m , which is the bin that overlaps with the right half of bin number k at Level 2. $m=\left\lfloor \frac{PV_{max}}{S}\right\rfloor$ +1 from lemma statement.

(Now,)

$$ k+m>m $$

From Eq. (19),

$$ {P}_{max}-\left(k\ast S\right)<\frac{S}{2}\kern0.75em \Rightarrow \kern0.75em \frac{\kern0.5em {P}_{max}-\frac{S}{2}}{S}<k $$

(20)

and from Eq. (18): $\left\lfloor \frac{P_{max}}{S}\right\rfloor +1=k+1$

$$ {\displaystyle \begin{array}{c}\Rightarrow {P}_{max}\ge \mathrm{k}\ast S\\ {}\Rightarrow {P}_{max}\hbox{--} \frac{S}{2}\ge k\ast S-\frac{S}{2}\\ {}\Rightarrow \frac{P_{max}\hbox{--} \frac{S}{2}}{S}\ge k-\frac{1}{2}>k-1\end{array}} $$

(21)

From Eq. (20) and Eq. (21)

$$ {\displaystyle \begin{array}{c}k-1<\frac{P_{max}\hbox{--} \frac{S}{2}}{S}\kern0.5em <k\\ {}\Rightarrow \left\lfloor \frac{P_{max}\hbox{--} \frac{S}{2}}{S}\right\rfloor =k-1\\ {}\Rightarrow \left\lfloor \frac{P_{max}\hbox{--} \frac{S}{2}}{S}\right\rfloor +1=k\end{array}} $$

(22)

From definition of (${\mathrm{B}}_k^s$)_ij in Eq. (12), we get:

$$ {\left({\mathrm{B}}_{k+m}^s\right)}_{\mathrm{XbYb}}=1 $$

(23)

Bin k has values ranging from (k − 1) ∗ S to k ∗ S and Bin k + m has the values $\left(k-1\right)\ast S+\frac{S}{2}\ to\ k\ast S+\frac{S}{2}$

$$ {\displaystyle \begin{array}{c}\left(k-1\right)\ast S\le Range\ of\ Bin\ k\le k\ast S\\ {}\left(k-1\right)\ast S+\frac{S}{2}\kern0.5em \le Range\ of\ Bin\ \left(k+m\right)\le k\ast S+\frac{S}{2}\\ {}\Rightarrow \left(\boldsymbol{k}-\mathbf{1}\right)\ast \boldsymbol{S}\mathbf{\le}\boldsymbol{Range}\ \boldsymbol{of}\ \boldsymbol{Bin}\ \boldsymbol{k}\cup \boldsymbol{Range}\ \boldsymbol{of}\ \boldsymbol{Bin}\ \left(\boldsymbol{k}+\boldsymbol{m}\right)<\boldsymbol{k}\ast \boldsymbol{S}+\frac{\boldsymbol{S}}{\mathbf{2}}\end{array}} $$

(24)

From definition of ${\Delta}_k^s$:

$$ {\Delta}_k^s={\mathrm{B}}_k^s\cup {\mathrm{B}}_{k+\left\lfloor \frac{PV_{max}}{S}\right\rfloor \kern0.5em +1}^s={\mathrm{B}}_k^s\cup {\mathrm{B}}_{k+m}^s $$

Thus, from Eq. (24)

$$ \left(\boldsymbol{k}-\mathbf{1}\right)\ast \boldsymbol{S}\le \boldsymbol{Range}\ \boldsymbol{of}\ {\boldsymbol{\Delta}}_{\boldsymbol{k}}^{\boldsymbol{s}}<\boldsymbol{k}\ast \boldsymbol{S}+\frac{\boldsymbol{S}}{\mathbf{2}} $$

(25)

Rearranging Eq. (19) we get

$$ {P}_{max}<k\ast S+\frac{S}{2} $$

(26)

And because P_min lies in Bin k from Eq. (14):

$$ \left(k-1\right)\ast S\le {P}_{min} $$

(27)

Now, any PixelValue in region R will lie between P_max and P_min.

From (26) and (27) we get:

$$ \left(\boldsymbol{k}-\mathbf{1}\right)\ast \boldsymbol{S}\le {\boldsymbol{P}}_{\boldsymbol{min}}\le \boldsymbol{PixelValue}\left(\boldsymbol{i},\boldsymbol{j}\right)\le {\boldsymbol{P}}_{\boldsymbol{max}}<\boldsymbol{k}\ast \boldsymbol{S}+\frac{\boldsymbol{S}}{\mathbf{2}} $$

(28)

From Eqs (25) and (28), we thus prove that ${\left({\boldsymbol{\Delta}}_{\boldsymbol{k}}^{\boldsymbol{s}}\right)}_{\boldsymbol{ij}}=\mathbf{1}$ for all (i, j) ∈ R

$$ \mathbf{Case}\ \mathbf{2.2}:{\displaystyle \begin{array}{c}{P}_{max}-\left(k\ast S\right)\ge \frac{S}{2}\\ {}{P}_{min}={P}_{max}-D\end{array}} $$

(29)

Since D ≤ S and ${P}_{min}\ge {P}_{max}-S\ge \left(k\ast S\right)+\frac{S}{2}-S=\left(k-1\right)\ast S+\frac{S}{2}$, hence,

$$ {P}_{min}\ge \left(k-1\right)\ast S+\frac{S}{2} $$

(30)

And from Eq. (14):

$$ k=\left\lfloor \frac{P_{min}}{S}\right\rfloor +1\Rightarrow {P}_{min}<k\ast S $$

(31)

Hence, from (28) and (29) we get:

$$ \Rightarrow \left(k-1\right)\ast S+\frac{S}{2}\le \kern0.5em {P}_{min}<k\ast S $$

(32)

$$ Subtracting\ \frac{S}{2} we\ get,\kern0.75em \left(k-1\right)\ast S+\frac{S}{2}-\frac{S}{2}\le \kern0.5em {P}_{min}-\frac{S}{2}<k\ast S-\frac{S}{2} $$

(33)

$$ \Rightarrow \kern0.5em \left(k-1\right)\kern0.5em \le \frac{P_{min}-\frac{S}{2}}{S}<k-\frac{1}{2}<k $$

(34)

$$ k-1\le \frac{P_{min}-\frac{S}{2}}{S}<k $$

(35)

$$ {\displaystyle \begin{array}{c}\Rightarrow \left\lfloor \frac{P_{min}-\frac{S}{2}}{S}\right\rfloor =k-1\\ {}\Rightarrow \left\lfloor \frac{P_{min}-\frac{S}{2}}{S}\right\rfloor +1=k\kern1em \Rightarrow \left\lfloor \frac{P_{min}-\frac{S}{2}}{S}\right\rfloor +1=\left(k+m\right)-m\end{array}} $$

(36)

From definition of (${\mathrm{B}}_k^s$)_ij: ${\left({\mathrm{B}}_{k+m}^s\right)}_{\mathrm{XbYb}}\kern0.75em =1$

Bin k + 1 has values ranging from k ∗ S to (k + 1) ∗ S and bin k + m has the values $\left(k-1\right)\ast S+\frac{S}{2}\ to\ k\ast S+\frac{S}{2}$

$$ k\ast S\le Range\ of\ Bin\ \left(k+1\right)\le \left(k+1\right)\ast S $$

$$ \left(k-1\right)\ast S+\frac{S}{2}\kern0.5em \le Range\ of\ Bin\ \left(k+m\right)\le k\ast S+\frac{S}{2} $$

$$ \Rightarrow \left(\boldsymbol{k}-\mathbf{1}\right)\ast \boldsymbol{S}+\frac{\boldsymbol{S}}{\mathbf{2}}\kern0.5em \le \boldsymbol{Range}\ \boldsymbol{of}\ \boldsymbol{Bin}\ \left(\boldsymbol{k}+\mathbf{1}\right)\kern0.5em \cup \kern0.5em \boldsymbol{Range}\ \boldsymbol{of}\ \boldsymbol{Bin}\ \left(\boldsymbol{k}+\boldsymbol{m}\right)\le \left(\boldsymbol{k}+\mathbf{1}\right)\ast \boldsymbol{S} $$

And from the definition of ${\Delta}_{k+m}^s$:

$$ {\Delta}_{k+m}^s={\mathrm{B}}_{k+m}^s\cup {\mathrm{B}}_{k+m-m+1}^s={\mathrm{B}}_{k+m}^s\cup {\mathrm{B}}_{k+1}^s $$

Thus,

$$ \left(\boldsymbol{k}-\mathbf{1}\right)\ast \boldsymbol{S}+\frac{\boldsymbol{S}}{\mathbf{2}}\le \boldsymbol{Range}\ \boldsymbol{of}\ {\boldsymbol{\Delta}}_{\boldsymbol{k}+\boldsymbol{m}}^{\boldsymbol{s}}\le \left(\boldsymbol{k}+\mathbf{1}\right)\ast \boldsymbol{S} $$

(37)

From Eq. (18) we have

$$ {P}_{max}\le k\ast S\le \left(k+1\right)\ast S $$

(38)

And P_min from Eq. (30) we get,

$$ \left(k-1\right)\ast S+\frac{S}{2}\le {P}_{min} $$

Now, any PixelValue in region R will lie between P_max and P_min

$$ \left(\boldsymbol{k}-\mathbf{1}\right)\ast \boldsymbol{S}+\frac{\boldsymbol{S}}{\mathbf{2}}\le {\boldsymbol{P}}_{\boldsymbol{min}}\le \boldsymbol{PixelValue}\left(\boldsymbol{i},\boldsymbol{j}\right)\le {\boldsymbol{P}}_{\boldsymbol{max}}<\left(\boldsymbol{k}+\mathbf{1}\right)\ast \boldsymbol{S} $$

(39)

From Eqs. (37) and (39), we thus prove that ${\left({\boldsymbol{\Delta}}_{\boldsymbol{k}}^{\boldsymbol{s}}\right)}_{\boldsymbol{ij}}=\mathbf{1}$ for all (i, j) ∈ R

Thus, for all the cases the lemma gets proved.