A robust approach to text line grouping in online handwritten Japanese documents

Abstract

In this paper, we present an effective approach for grouping text lines in online handwritten Japanese documents by combining temporal and spatial information. With decision functions optimized by supervised learning, the approach has few artificial parameters and utilizes little prior knowledge. First, the strokes in the document are grouped into text line strings according to off-stroke distances. Each text line string, which may contain multiple lines, is segmented by optimizing a cost function trained by the minimum classification error (MCE) method. At the temporal merge stage, over-segmented text lines (caused by stroke classification errors) are merged with a support vector machine (SVM) classifier for making merge/non-merge decisions. Last, a spatial merge module corrects the segmentation errors caused by delayed strokes. Misclassified text/non-text strokes (stroke type classification precedes text line grouping) can be corrected at the temporal merge stage. To evaluate the performance of text line grouping, we provide a set of performance metrics for evaluating from multiple aspects. In experiments on a large number of free form documents in the Tokyo University of Agriculture and Technology (TUAT) Kondate database, the proposed approach achieves the entity detection metric (EDM) rate of 0.8992 and the edit-distance rate (EDR) of 0.1114. For grouping of pure text strokes, the performance reaches EDM of 0.9591 and EDR of 0.0669.

Introduction

With the increasing use of tablet PCs, electronic whiteboards, and digital pens on paper, users can draw various heterogeneous structures such as text, drawings and table forms freely on a large writing area. Such freely handwritten ink documents bring new challenges to automatic analysis and recognition. An ink document (online handwritten document) comprises a sequence of strokes, which are to be grouped into various structural units (drawings, text lines, words, characters, etc). Text lines are the most salient structures in ink documents and their reliable extraction is the pre-condition to further processing tasks such as character recognition, text editing and retrieval. This paper addresses the problem of text line grouping in freeform online handwritten Japanese documents.

Earlier systems of ink document analysis [1], [2] used to assume horizontal text lines and simply separate the text lines according to the projection profile. Liwichi et al. [3] proposed a text line detection method based on dynamic programming (DP) with a cost function involving hand-tuned parameters. This system was designed for ink pages with parallel text lines, and the online information was used merely in post-processing. It is important to point out that the assumption of text line regularity is often violated in ink documents because of the arbitrary line directions and inter-line distances [4]. Nevertheless, compared to offline documents, online handwriting has the advantage that the temporal order of strokes is available, which provides useful cues for text line grouping besides spatial information. Higher accuracy of grouping is achievable by utilizing both the temporal and spatial information.

The system developed by Shilman et al. [4] over-segments the stroke sequence by DP with a cost function reflecting the confidence that a given set of strokes belongs to one word, and the text lines are grouped by merging pairs of stroke clusters in aggressive steps. Although the system utilizes good heuristics in merging, it lacks a principled framework to guarantee the optimality.

Ye et al. [5] proposed a global optimization method integrating the likelihood of the resulting lines and the consistency of their configuration for text line grouping. The initial segmentation by DP minimizes a cost function involving the number of text lines as a constraint, which can effectively avoid over-segmentation. Then, a local gradient-decent algorithm is used to evaluate the splitting and merging hypotheses iteratively to minimize the global cost function. The global optimization method is superior to heuristic rules because it incorporates both the prior knowledge and the information from data in a principled formulation. However, without a training procedure, the parameters in the cost function have to be tuned manually. In their enhanced system [6], a high-confidence-first (HCF) method was proposed to group text lines and writing regions, with an AdaBoost classifier for making merge/non-merge decisions. The classifier was trained from ink data and provides confidence of merge.

For Japanese documents, Nakagawa et al. [7] describe a system in which the stroke sequence is segmented into text lines according to off-stroke (pen-up) distances and changing of writing directions based on the fact that off-strokes within a text line are mostly shorter than those between text lines and the text lines are usually straight. However, due to the variability of character size and spacing, this segmentation method does not perform reliably.

To better utilize the temporal and spatial information in online handwriting, we propose a text line grouping approach with very few artificial parameters and little prior knowledge. We only assume that the text lines are approximately straight (curved lines will be segmented as piecewise linear text lines by the proposed approach), while the writing directions can be arbitrary and need not to be parallel with each other. Unlike the projection based methods [1], [2], we do not estimate the inter-line distance, so the text lines can be arbitrarily close to each other. The above merits also favor the extensibility of the proposed method for text line grouping in non-Japanese ink documents.

The proposed text line grouping process performs in several stages. Initially, strokes are grouped into text line strings according to off-stroke distances. A discriminant function is trained under the string-level minimum classification error (MCE) criterion [8], [9] to separate the over-merged text lines with the beam search strategy. To correct the stroke classification errors and merge the misclassified strokes to text lines in documents of mixed text/drawing (in this case, a text/non-text classification procedure precedes text line grouping), an support vector machine (SVM) classifier is trained to make merge/non-merge decisions. Last, the errors caused by delayed strokes are amended by a spatial merge step.

Partitioning stroke sequence into text lines based on optimization has been tried by previous works [3], [4], [5], but without a training procedure, the parameters in the cost function have to be tuned manually. We take into account the similarity of this problem to character string recognition integrating character segmentation and classification [10], [11] and use string-level MCE training [8], [9], which has been applied to numeral string recognition [12], for training the parameters of cost function. The merge/non-merge decision by an SVM classifier in temporal merge stage is similar to that of Ref. [6], which uses an AdaBoost classifier. By our temporal merge module, text/non-text stroke classification errors can be corrected.

As for performance evaluation of text line grouping algorithms, there has not been a unified criterion. In Ref. [5], the recall metric, which is defined as the number of correct lines divided by the number of labeled lines in each page, is used to measure the accuracy of the system. In Refs. [4] and [6], the edit-distance metrics are employed to evaluate the system performance. The edit-distance between the text line detection result and the ground-truth is defined as the minimum number of split or merge operations needed to correct all errors, and the error rate is defined as the total number of edit operations divided by the total number of labeled text lines [6]. The system proposed by Liwichi et. al. [3] adopts the stroke classification rate defined as the number of correctly assigned strokes divided by the total number of strokes, and the document classification rate defined as the number of correctly processed documents divided by the total number of documents. Each of the above metrics can evaluate the system from a certain aspect, while to achieve an overall evaluation, we need a systematic methodology. Inspired by the performance evaluation methods for graphics recognition systems [13], which are also used in the ICDAR page segmentation competitions and offline handwriting segmentation contest [14], we give an extensive set of evaluation metrics.

To demonstrate the effectiveness of the proposed approach, we have experimented on the Tokyo University of Agriculture and Technology (TUAT) Kondate database [7] with two settings: one with perfect text/non-text separation (stroke type labels given) and the other with a stroke classification module. The results show that the proposed text line grouping method is robust for both the two cases.

This paper is an extension to a conference paper [15]. The extension is in several respects: more details of techniques description, improved features in MCE training, extensive performance metrics, experimental results and discussions. The rest of this paper is organized as follows: Section 2 gives an overview of our ink document analysis system. Section 3 details the text line grouping approach. Section 4 describes the performance evaluation metrics. Section 5 presents the experimental results and Section 6 offers our concluding remarks.