Unsupervised language model adaptation for handwritten Chinese text recognition

Highlights

• We propose an unsupervised language model (LM) adaptation framework for handwritten Chinese text recognition.

• We use a two-pass recognition strategy with a pre-defined multi-domain LM set.

• The adaptive LM is dynamically generated by three methods of model selection, model combination and model reconstruction.

• We compress the LM set by split vector quantization and principal component analysis.

Abstract

This paper presents an effective approach for unsupervised language model adaptation (LMA) using multiple models in offline recognition of unconstrained handwritten Chinese texts. The domain of the document to recognize is variable and usually unknown a priori, so we use a two-pass recognition strategy with a pre-defined multi-domain language model set. We propose three methods to dynamically generate an adaptive language model to match the text output by first-pass recognition: model selection, model combination and model reconstruction. In model selection, we use the language model with minimum perplexity on the first-pass recognized text. By model combination, we learn the combination weights via minimizing the sum of squared error with both L2-norm and L1-norm regularization. For model reconstruction, we use a group of orthogonal bases to reconstruct a language model with the coefficients learned to match the document to recognize. Moreover, we reduce the storage size of multiple language models using two compression methods of split vector quantization (SVQ) and principal component analysis (PCA). Comprehensive experiments on two public Chinese handwriting databases CASIA-HWDB and HIT-MW show that the proposed unsupervised LMA approach improves the recognition performance impressively, particularly for ancient domain documents with the recognition accuracy improved by 7 percent. Meanwhile, the combination of the two compression methods largely reduces the storage size of language models with little loss of recognition accuracy.

Introduction

Handwritten Chinese character recognition has attracted a lot of attention since the 1970s and has achieved tremendous advances [1], [2]. However, the recognition of unconstrained handwritten Chinese texts has been reported only in recent years, and the reported accuracies are quite low (e.g., character-level correct rate of 39 percent in [3]). Besides the divergence of writing styles, handwritten text recognition is difficult due to the weak lexical constraint: the number of sentence classes is infinite. Although our recent work by integrating multiple contexts including linguistic context achieved a high correct rate of 91 percent [4], there are still many recognition errors due to the insufficient modeling of linguistic context, particularly, the language model often mismatches the domain of handwritten text. To deal with this mismatch problem, we investigate language model adaptation for handwritten Chinese text recognition (HCTR) in this paper, particularly, unsupervised adaptation for the scenario that there is no prior information about the domain of text.

Language model (LM) provides a principled way to quantify the uncertainties associated with the natural language, but this uncertainty is variable in the texts of different domains. The modeling of diverse domains was originally studied in speech recognition [5], leading to the research of language model adaptation. The researchers often combine a generic LM with a domain-specific LM that is more relevant to the recognition task. However, it is usually difficult to get enough domain-specific data to learn the domain-specific model [5], and a growing interest is evident in collecting texts from the Internet to supplement sparse domain-specific resources [6]. For Chinese resources, the Sogou Labs¹ provide a large set of resources relevant to diverse domains extracted from the Internet, which can be used to train a multi-domain LM set via the SRILM toolkit [7]. This LM set can be adaptively applied in HCTR according to the domain of each document.

Language model adaptation (LMA) is difficult when the domain of recognition task is unknown a priori, which can only be solved by unsupervised approaches. Many efforts have been made in this direction in speech recognition. One method is to calculate the probability of one word given a document (herein, the history of recognized text) as an uni-gram model by latent topic analysis [5], [8], [9], [10], then interpolate it with the generic LM. Another popular method is to use the recognized text directly to estimate an adaptive n-gram model [11], [12], which is interpolated with the generic LM for the next-pass recognition. This method is usually based on a multi-pass (e.g., two-pass) recognition framework. In handwriting recognition, unsupervised LMA has been reported only in recent years, and the adaptive LM is very simple. For example, Xiu and Baird [13] applied the multi-pass recognition strategy iteratively to adapt a word lexicon, and Lee and Smith [14] iteratively modified uni-gram probabilities further for English whole-book recognition, where the texts are very long.

Another difficulty of LMA is that the transcripts of handwritten texts are usually short. There are usually only a few hundreds of characters in a handwritten text (e.g., Fig. 1), impeding the direct adaption of the lexicon or n-gram probabilities. In such situation, the interpolation of models from a pre-defined multi-domain LM set is usually used in speech recognition, and the interpolation weights are learned by the maximum a posteriori (MAP) estimation from the held-out data similar to the task domain (supervised LMA, e.g., [5], [15]), or the recognized texts (unsupervised LMA, e.g., [16]) or both (e.g., [17]). This solution also brings a problem of the large storage size of the LM set. To overcome this problem, the size of each LM is usually reduced by entropy-based pruning [18] and quantization-based compression [19].

In this paper, we propose an unsupervised LMA framework for HCTR. For no prior information of the document to recognize (test document) is available, we use a two-pass recognition strategy. In consideration of various and short handwritten texts, we dynamically generate an adaptive LM to match the first-pass recognized text of each document using a pre-defined multi-domain LM set. We propose three methods to generate adaptive LMs, namely model selection, model combination and model reconstruction. The model selection method is to select the best LM according to the minimum perplexity criterion. In model combination, we estimate the combination weights by minimizing the sum of squared error (MSE) with both L2-norm and L1-norm regularization. By model reconstruction, the adaptive LM is constructed by a group of orthogonal bases. To make the adaption approach practical, we also consider the reduction of computational cost and storage space. To speed up the two-pass recognition, we store the candidate character classes of each candidate character pattern in the first-pass recognition to avoid repeated classification in the second-pass. To reduce the storage size of LM set, we compress the LMs using split vector quantization (SVQ) and principal component analysis (PCA). Finally, we evaluated the recognition performance on two public Chinese handwriting databases CAISA-HWDB [20] and HIT-MW [21], and showed large improvement of performance gained by the proposed unsupervised LMA methods with comparable computational cost to the baseline system.

Unlike previous works on unsupervised LMA in speech recognition (e.g., [17]), we regard the language model combination as a linear regression problem. We learn combination weights via minimizing an error cost function (i.e., MSE) and further consider model sparsity by adding L1-norm regularization, which is totally different from the MAP or MBR (minimum Bayes' risk) based framework [17]. The MAP estimation is based on the perplexity and MBR takes into account the acoustic model under supervised framework, and both get a local optimal solution by Baum–Welch algorithm, while the MSE-based method aims at error loss minimization on the recognized text with a global optimal solution. Another contribution of this paper is the new LMA method based on model reconstruction by applying PCA to a pre-defined LM set. This idea is motivated by a technique developed by Sirovich and Kirby [22] for efficiently representing images of human faces using PCA, and it has also been successfully used in image processing like active shape models [23]. More importantly, the focus of this work is to investigate the role of LMA in Chinese handwriting recognition, which, to our best of knowledge, has not been investigated in depth in handwriting recognition field. A preliminary conference version of the paper was presented in [24], and this extended version provides more detailed descriptions, presents additional LMA methods and a significantly extended experimental validation. By customizing the handwriting recognition algorithm, the proposed approach can also apply to the recognition of the documents of other languages (such as English and Arabic).

The rest of this paper is organized as follows: Section 2 reviews some related works; Section 3 gives an overview of our HCTR system with unsupervised LMA; Section 4 briefly describes the statistical language models used in this paper; Section 5 describes in details the unsupervised LMA methods of model selection, model combination and model reconstruction; Section 6 introduces the LM compression methods; Section 7 presents the experimental results, and Section 8 draws concluding remarks.