Abstract
Word segmentation is an essential process in Chinese information processing. Although related researches were reported and made progresses, the Unknown Named Entity (UNE) problem in segmentation is not fully solved. This usually degrades the accuracy of segmentation in general. In this paper, a model to identify UNEs for improving the overall performance of the segmentation is presented. In order to capture the NE information, functions of characters or words are defined with tags. In addition, useful surrounding contexts are collected from a corpus and used as features. The model is constructed based on Maximum Entropy to handle the UNE identification as tagging problem. Empirical experiments show that the overall accuracy of the segmentation is improved after integrating the UNE identification module into the word segmenter.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Leong, K.S., Wong, F., Li, Y.P., Dong, M.C.: Chinese Word Boundaries Detection Based on Maximum Entropy Model. In: Proceedings of the 11th International Conference on Enhancement and Promotion of Computational Methods in Engineering and Science (EPMESC-XI), Kyoto, Japan (2007)
Leong, K.S., Wong, F., Tang, C.W., Dong, M.C.: CSAT: A Chinese Segmentation and Tagging Module Based on the Interpolated Probabilistic Model. In: Proceedings of the 10th International Conference on Enhancement and Promotion of Computational Methods in Engineering and Science (EPMESC-X), Sanya, Hainan, China, pp. 1092–1098 (2006)
Lv, Y., Li, T., Yang, M., Yu, H., Li, S.: Leveled Unknown Chinese Words Resolution by Dynamic Programming. Journal of Chinese Information Processing 15(1), 28–33 (2000)
Sun, J., Gao, J., Zhang, L., Zhou, M., Huang, C.: Chinese Named Entity Identification Using Class-Based Language Model. In: Proceedings of the 19th International Conference on Computational Linguistics, Taipei, Taiwan, pp. 1–7 (2002)
Liu, Q., Zhang, H.P., Yu, H.K., Cheng, X.Q.: Chinese Lexical Analysis Using Cascaded Hidden Markov Model. Journal of Chinese Information Processing 41(8), 1421–1429 (2004)
Borthwick, A.: A Maximum Entropy Approach to Named Entity Recognition. PhD thesis, New York University (1999)
Xue, N., Shen, L.: Chinese Word Segmentation as LMR Tagging. In: Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing, Sapporo, Japan, pp. 176–179 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Leong, K.S., Wong, F., Li, Y., Dong, M.C. (2008). Integration of Named Entity Information for Chinese Word Segmentation Based on Maximum Entropy. In: Huang, DS., Wunsch, D.C., Levine, D.S., Jo, KH. (eds) Advanced Intelligent Computing Theories and Applications. With Aspects of Theoretical and Methodological Issues. ICIC 2008. Lecture Notes in Computer Science, vol 5226. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-87442-3_118
Download citation
DOI: https://doi.org/10.1007/978-3-540-87442-3_118
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-87440-9
Online ISBN: 978-3-540-87442-3
eBook Packages: Computer ScienceComputer Science (R0)