Transform invariant text extraction

Abstract

Automatically extracting texts from natural images is very useful for many applications such as augmented reality. Most of the existing text detection systems require that the texts to be detected (and recognized) in an image are taken from a nearly frontal viewpoint. However, texts in most images taken naturally by a camera or a mobile phone can have a significant affine or perspective deformation, making the existing text detection and the subsequent OCR engines prone to failures. In this paper, based on stroke width transform and texture invariant low-rank transform, we propose a framework that can detect and rectify texts in arbitrary orientations in the image against complex backgrounds, so that the texts can be correctly recognized by common OCR engines. Extensive experiments show the advantage of our method when compared to the state of art text detection systems.

Appendix: Detailed derivation of the ADM for multi-component TILT

1.1 A.1 ALM and ADM methods

The alternating direction method (ADM, also called inexact augmented Lagrange multiplier (IALM) method in [29]) is a variant of the augmented Lagrange multiplier (ALM) method. The usual ALM method for solving the following model problem:

$$ \min f(x),\quad\mbox{s.t.}\quad h(x)=0, $$

(9)

where $f:\mathbb{R}^{n}\rightarrow \mathbb{R}$ and $h:\mathbb{R}^{n} \rightarrow \mathbb{R}^{m}$, is to minimize over its augmented Lagrangian function:

$$ L(x,y,\mu) = f(x) + \bigl\langle y, h(x)\bigr\rangle + \frac{\mu}{2}\bigl\|h(x)\bigr\|_F^2, $$

(10)

where μ is a positive scalar and $y\in \mathbb{R}^{m}$ is the Lagrange multiplier. The ALM method is outlined as Algorithm 1 (see [36] for more details).

Algorithm 1

(General Augmented Lagrange Multiplier Method)

Solving the subproblem

$$\min_{x} L(x,y_k,\mu_k) $$

is actually not always easy. When f(x) is separable, e.g., f(x)=f ₁(a)+f ₂(b), where x=(a ^T,b ^T)^T, we may solve this subproblem by minimizing over a and b alternately until convergence. However, in the case that f(x) is convex and h(x) is a linear function of x, the ADM requires updating a and b only once and it can still be proven that the iteration converges to the optimal solution [37]. The ADM for two variables is outlined as Algorithm 2. The generalization to the multi-variable and multi-constraint case is straightforward.

Algorithm 2

(General Alternating Direction Method)

1.2 A.2 Subproblems of multi-component TILT

When applying ADM to the linearized multi-component TILT, whose augmented Lagrangian function is

$$\begin{aligned} \begin{aligned} &L\bigl(\{A_i\},A,E,\Delta \tau,\{Y_i \},Y,\mu \bigr) \\ &\quad = \sum_{i = 1}^n {{\Vert {{A_i}} \Vert }_*} + \Vert A \Vert _* + \lambda \Vert E \Vert _1 \\ &\qquad {}+ \bigl\langle {Y,D \circ \tau + J\Delta \tau - A - E} \bigr\rangle + \sum _{i = 1}^n \bigl\langle {{Y_i},{A_i}- \mathcal{P}_i(A)} \bigr\rangle \\ &\qquad {}+ \frac{\mu }{2} \Biggl( \Vert {D \circ \tau + J\Delta \tau -A- E} \Vert _F^2 \\ &\qquad {}+ \sum_{i = 1}^n {\bigl \Vert {{A_i} -\mathcal{P}_i(A)} \bigr \Vert _F^2} \Biggr), \end{aligned} \end{aligned}$$

where $\mathcal{P}_{i}(A)$ extracts the ith block of A, we have to solve multiple subproblems:

$$\begin{aligned} A_i^{(k+1)} =&\arg\min_{A_i} L \bigl(A_1^{(k+1)},\ldots,A_{i-1}^{(k+1)},A_i, \\ &{}A_{i+1}^{(k)},\ldots, A_n^{(k)},A^{(k)},E^{(k)}, \Delta \tau^{(k)}, \\ &{}\bigl\{Y_j^{(k)}\bigr\},Y^{(k)}, \mu^{(k)}\bigr),\quad i=1,\ldots,n, \end{aligned}$$

(11)

$$\begin{aligned} A^{(k+1)} =& \arg\min_{A} L\bigl(\bigl \{A_j^{(k+1)}\bigr\},A,E^{(k)},\Delta \tau^{(k)}, \\ &{}\bigl\{Y_j^{(k)}\bigr\},Y^{(k)}, \mu^{(k)}\bigr), \end{aligned}$$

(12)

$$\begin{aligned} E^{(k+1)} =& \arg\min_{E} L\bigl(\bigl \{A_j^{(k+1)}\bigr\},A^{(k+1)},E,\Delta \tau^{(k)}, \\ &{}\bigl\{Y_j^{(k)}\bigr\},Y^{(k)}, \mu^{(k)}\bigr), \end{aligned}$$

(13)

$$\begin{aligned} \Delta \tau^{(k+1)} =& \arg\min_{E} L\bigl(\bigl \{A_j^{(k+1)}\bigr\},A^{(k+1)},E^{(k+1)}, \\ &{}\Delta \tau,\bigl\{Y_j^{(k)}\bigr\},Y^{(k)}, \mu^{(k)}\bigr). \end{aligned}$$

(14)

When solving (11), by leaving out the terms in L that are independent of A _i , we see

$$\begin{aligned} A_i^{(k+1)} =&\arg\min_{A_i} \|A_i\|_*+ \bigl\langle Y_i^{(k)},{A_i} - \mathcal{P}_i\bigl(A^{(k)}\bigr) \bigr\rangle \\ &{}+\frac{\mu^{(k)}}{2}\bigl \Vert {A_i} -\mathcal{P}_i \bigl(A^{(k)}\bigr) \bigr \Vert _F^2 \\ =&\arg\min_{A_i} \|A_i\|_*+\frac{\mu^{(k)}}{2} \bigl \Vert {A_i} - \hat{A}_i^{(k+1)}\bigr \Vert _F^2, \end{aligned}$$

where

$$\hat{A}_i^{(k+1)} = \mathcal{P}_i \bigl(A^{(k)}\bigr) - \bigl(\mu^{(k)}\bigr)^{-1}Y_i^{(k)}. $$

Then by Theorem 2.1 of [38], $A_{i}^{(k+1)}$ can be obtained by thresholding the singular values of $\hat{A}_{i}^{(k+1)}$. Namely, if the singular value decomposition (SVD) of $\hat{A}_{i}^{(k+1)}$ is $U_{ik}\varSigma_{ik}V_{ik}^{T}$, then

$$A_i^{(k+1)} = U_{ik}\mathcal{S}_{(\mu^{(k)})^{-1}}( \varSigma_{ik})V_{ik}^T, $$

where

$$\mathcal{S}_{\varepsilon}(x)=\max\bigl(|x|-\varepsilon,0\bigr)\operatorname {sgn}(x) $$

is the shrinkage operator.

Similarly, when solving (12) with some elaboration we have

$$\begin{aligned} A^{(k+1)} =&\arg\min_{A} \|A\|_* \\ &{} + \bigl\langle {Y^{(k)},D \circ \tau + J\Delta \tau^{(k)} - A - E^{(k)}} \bigr\rangle \\ & {}+ \sum_{i = 1}^n \bigl\langle {Y_i^{(k)}},{A_i^{(k+1)}} - \mathcal{P}_i(A) \bigr\rangle \\ & {}+ \frac{\mu^{(k)}}{2} \Biggl( \bigl \Vert D \circ \tau + J\Delta \tau^{(k)} - A - E^{(k)} \bigr \Vert _F^2 \\ &{}+ \sum_{i = 1}^n \bigl \Vert {A_i^{(k+1)}} -\mathcal{P}_i(A) \bigr \Vert _F^2 \Biggr) \\ =&\arg\min_{A} \|A\|_*+\mu^{(k)}\bigl \Vert A- \hat{A}^{(k+1)}\bigr \Vert _F^2, \end{aligned}$$

where

$$\begin{aligned} \hat{A}^{(k+1)} = &\frac{1}{2} \bigl(D \circ \tau + J\Delta \tau^{(k)} - E^{(k)}+\bigl(\mu^{(k)}\bigr)^{-1}Y^{(k)} \\ &{}+\tilde{A}^{(k+1)} \bigr), \end{aligned}$$

in which $\tilde{A}^{(k+1)}$ is a matrix whose ith block is $A_{i}^{(k+1)}+(\mu^{(k)})^{-1}Y_{i}^{(k)}$. Then again we can apply Theorem 2.1 of [38] to solve for A ^(k+1) as we did above for $A_{i}^{(k+1)}$.

When solving (13), we have

$$\begin{aligned} E^{(k+1)} =&\arg\min_{E} \lambda {\Vert E \Vert _1} \\ &{}+ \bigl\langle {Y^{(k)},D \circ \tau + J\Delta \tau^{(k)} - A^{(k+1)} - E} \bigr\rangle \\ &{}+\frac{\mu^{(k)}}{2}\bigl \Vert {D \circ \tau + J\Delta \tau^{(k)} - A^{(k+1)} - E} \bigr \Vert _F^2 \\ =&\lambda \Vert E \Vert _1 +\frac{\mu^{(k)}}{2}\bigl \Vert E - \hat{E}^{(k+1)}\bigr \Vert _F^2, \end{aligned}$$

where

$$\hat{E}^{(k+1)}= D \circ \tau + J\Delta \tau^{(k)} - A^{(k+1)}+\bigl(\mu^{(k)}\bigr)^{-1}Y^{(k)}. $$

Then its well known [39] that the solution to the above problem is

$$E^{(k+1)} = \mathcal{S}_{\lambda(\mu^{(k)})^{-1}}\bigl(\hat{E}^{(k+1)}\bigr), $$

where $\mathcal{S}$ is the shrinkage operator defined above.

When solving (14), we have

$$\begin{aligned} \Delta \tau^{(k+1)}= \arg\min_{\Delta \tau} \bigl\|J\Delta \tau- \widehat{\Delta \tau}^{(k+1)} \bigr\|_F^2, \end{aligned}$$

(15)

where

$$\widehat{\Delta \tau}^{(k+1)} = -D \circ \tau + A^{(k+1)} + E^{(k+1)} - \bigl(\mu^{(k)}\bigr)^{-1}Y^{(k)}. $$

So

$$\begin{aligned} \begin{array}{c} \Delta \tau^{(k+1)}= J^{\dagger}\widehat{\Delta \tau}^{(k+1)}, \end{array} \end{aligned}$$

where J ^† is the pseudo-inverse of J and $\widehat{\Delta \tau}^{(k+1)}$ is rearranged into a vector.

1.3 A.3 The complete algorithm

Now we can present the ADM for multi-component TILT as Algorithm 3.

Algorithm 3

(Solving Multi-Component TILT by ADM)

Note that in Algorithm 3, the transformed image D∘τ has to be normalized to prevent the region of interest from shrinking into a black pixel [2]. For more details of implementation issues, such as maintaining the area and aspect ratio, please refer to [2]. We will also post our code online if the paper is accepted.