Summary
|
For Full-Text PDF, please login, if you are a member of IEICE, or go to Pay Per View on menu list, if you are a nonmember of IEICE. |
|
Pruning-Based Unsupervised Segmentation for Korean In-Su KANG Seung-Hoon NA Jong-Hyeok LEE Publication IEICE TRANSACTIONS on Information and Systems Vol.E89-D No.10 pp.2670-2677 Publication Date: 2006/10/01 Online ISSN: 1745-1361 DOI: 10.1093/ietisy/e89-d.10.2670 Print ISSN: 0916-8532 Type of Manuscript: PAPER Category: Natural Language Processing Keyword: compound noun segmentation, unsupervised method, pruning technique, segmentation evaluation, Full Text: PDF(487KB)>>
Summary: Compound noun segmentation is a key component for Korean language processing. Supervised approaches require some types of human intervention such as maintaining lexicons, manually segmenting the corpora, or devising heuristic rules. Thus, they suffer from the unknown word problem, and cannot distinguish domain-oriented or corpus-directed segmentation results from the others. These problems can be overcome by unsupervised approaches that employ segmentation clues obtained purely from a raw corpus. However, most unsupervised approaches require tuning of empirical parameters or learning of the statistical dictionary. To develop a tuning-less, learning-free unsupervised segmentation algorithm, this study proposes a pruning-based unsupervised technique that eliminates unhelpful segmentation candidates. In addition, unlike previous unsupervised methods that have relied on purely character-based segmentation clues, this study utilizes word-based segmentation clues. Experimental evaluations show that the pruning scheme is very effective to unsupervised segmentation of Korean compound nouns, and the use of word-based prior knowledge enables better segmentation accuracy. This study also shows that the proposed algorithm performs competitively with or better than other unsupervised methods. | ||||||||||
| ||||||||||
