Abstract
The lack of word delimiters such as spaces in Chinese texts makes word segmentation a special issue in Chinese text processing. As the volume of Chinese texts grows rapidly on the Internet, the number of unknown words increases accordingly. However, word segmentation approaches relying solely on existing dictionaries are helpless in handling unknown words. In this paper, we propose a novel unsupervised method to segment large Chinese corpora using contextual information. In particular, the number of characters preceding and following a string, known as the accessors of the string, is used to measure the independence of the string. The greater the independence, the more likely it is that the string is a word. The segmentation problem is then considered an optimization problem to maximize the target function of this number over all word candidates in an utterance. Our purpose here is to explore the best function in terms of segmentation performance. The performance is evaluated with the word token recall measure in addition to word type precision and word type recall. Among the three types of target functions that we have explored, polynomial functions turn out to outperform others. This simple method is effective in unsupervised segmentation of Chinese texts and its performance is highly comparable to other recently reported unsupervised segmentation methods.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ando, R.K., Lee, L.: Mostly-unsupervised statistical segmentation of Japanese: Applications to Kanji. NAACL-1, 241–248 (2000)
Brent, M.R.: An efficient, probabilistically sound algorithm for segmentation and word discovery. Machine Learning 34, 71–106 (1999)
Brent, M.R., Tao, X.: Chinese text segmentation with MBDP-1: Making the most of training corpora. In: ACL-EACL 2001, Toulouse, France, pp. 82–89 (2001)
Cheng, K.-S., Young, G.H., Wong, K.-F.: A study on word-based and integral-bit Chinese text compression algorithm. Journal of the American Society for Information Science 50, 218–228 (1999)
Dai, Y., Khoo, C., Loh, T.E.: A new statistical formula for Chinese text segmentation incorporating contextual information. In: SIGIR 1999, pp. 82–89 (1999)
Douglas, B., Berger, A., Lafferty, J.D.: Statistical models for text segmentation. Machine Learning 34(1-3), 177–210 (1999)
Harris, Z.S.: Papers in Structual and Transformational Linguistics, pp. 68–77. D. Reidel, Dordrecht (1970)
Huang, J., Powers, D.: Experiments on unsupervised Chinese word segmentation and classification. In: First Student Workshop on Computational Linguistics, Peking University, pp. 83–89 (2002)
Kit, C., Liu, Y., Liang, N.: On methods of Chinese automatic word segmentation. Journal of Chinese Information Processing 3(1), 1–32 (1989)
Liang, N.: CDWS – An automatic word segmentation system for written Chinese. Journal of Chinese Information Processing 1(2), 44–52 (1986)
Peng, F., Schuurmans, D.: Self-supervised Chinese word segmentation. In: Advances in Intelligent Data Analysis (IDA 2001), Cascais, Portugal, pp. 238–247 (2001)
Ponte, J.M., Bruce, W.C.: Useg: a retargetable word segmentation procedure for information retrieval. Technical Report TR96-2. University of Massachusetts (Amherst, MA (1996)
Sproat, R., Shih, C., Gale, W., Chang, N.: A stochastic finitestate word-segmentation algorithm for Chinese. Computational Linguistics 22(3), 377–404 (1996)
Sun, M., Shen, D., Tsou, B.K.: Chinese word segmentation without using lexicon and hand-crafted training data. In: COLING-ACL 1998 (II), Montreal, pp. 1265–1271 (1998)
Teahan, W.J., Wen, Y., McNab, R.J., Witten, I.H.: A compression-based algorithm for Chinese word segmentation. Computational Linguistics 26(3), 375–393 (2000)
Wang, Y.-H., Su, H.-J., Mo, Y.: Automatic processing of Chinese words. Journal of Chinese Information Processing 4(4), 1–11 (1990)
Wu, Z., Tseng, G.: Chinese text segmentation for text retrieval: Achievements and problems. Journal of the American Society for Information Sceince 44(9), 532–542 (1993)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Feng, H., Chen, K., Kit, C., Deng, X. (2005). Unsupervised Segmentation of Chinese Corpus Using Accessor Variety. In: Su, KY., Tsujii, J., Lee, JH., Kwong, O.Y. (eds) Natural Language Processing – IJCNLP 2004. IJCNLP 2004. Lecture Notes in Computer Science(), vol 3248. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30211-7_73
Download citation
DOI: https://doi.org/10.1007/978-3-540-30211-7_73
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-24475-2
Online ISBN: 978-3-540-30211-7
eBook Packages: Computer ScienceComputer Science (R0)