Abstract
As a sequence of two or more consecutive individual words inherent with contextual semantics of individual words, multi-word attracts much attention from statistical linguistics and of extensive applications in text mining. In this paper, we carried out a series studies on multi-word extraction from Chinese documents. Firstly, we proposed a new statistical method, augmented mutual information (AMI), for words’ dependency. Experiment results demonstrate that AMI method can produce a recall on average as 80% and its precision is about 20%-30%. Secondly, we attempt to utilize the variance of occurrence frequencies of individual words in a multi-word candidate to deal with the rare occurrence problem. But experimental results cannot validate the effectiveness of variance. Thirdly, we developed a syntactic method based on lexical regularities of Chinese multi-word to extract the multi-words from Chinese documents. Experimental results demonstrate that this syntactical method can produce a higher precision on average as 0.5521 than AMI method but it cannot produce a comparable recall. Finally, the possible breakthrough on combining statistical methods and syntactical methods is shed light on.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Firth, J.R.: A Synopsis of Linguistic Theory 1930-1955. Studies in Linguistic Analysis. Philological Society. Blackwell, Oxford (1957)
Justeson, J.S., Katz, S.M.: Technical terminology: some linguistic properties and an algorithm for identification in text. Natural Language Engineering 1(1), 9–27 (1995)
Bourigault, D.: Surface Grammatical Analysis for the Extraction of Terminological Noun Phrases. In: Proceedings of the 14th International Conference on Computational Linguistics, Nantes, France, pp. 977–981 (1992)
Kupiec, J.: MURAX: A robust linguistic approach for question answering using an on-line encyclopedia. In: Proceedings of the Sixteenth Annual International ACM Conference on Research and Development in Information Retrieval, Pittsburgh, PA, USA, June 27 - July 1, 1993, pp. 181–190 (1993)
Smadja, F.: Retrieving collocations from text: Xtract. Computational Linguistics 19(1), 143–177 (1993)
Church, K.W., Robert, L.M.: Introduction to special issue on computational linguistics using large corpora. Computational Linguistics 19(1), 1–24 (1993)
Chen, J.S., et al.: Identifying multi-word terms by text-segments. In: Proceedings of the seventh international conference on Web-Age information Management Workshops (WAIMV 2006), HongKong, pp. 10–19 (2006)
Park, Y.J., et al.: Automatic Glossary Extraction: Beyond Terminology Identification. In: Proceedings of the 19th international conference on Computational linguistics, Taiwan, pp. 1–17 (2002)
Church, K.W., Hanks, P.: Word association norms, mutual information, and lexicography. Computational Linguistics 16(1), 22–29 (1990)
Sproat, R., Shinh, C.: A statistic method for finding word boundaries in Chinese text. Computer Processing of Chinese and Oriental Language 4(4), 336–351 (1990)
Yamamoto, M., Church, K.W.: Using suffix arrays to compute term frequency and document frequency for all substrings in a corpus. Computational Linguistics 27(1), 1–30 (2001)
Kita, K., et al.: A comparative study of automatic extraction of collocations from Corpora: mutual information vs. cost criteria. Journal of Natural Language Processing 1(1), 21–29 (1992)
Chen, B.X., Du, L.M.: Preparatory work on automatic extraction of bilingual multi-word units from parallel corpora. Computational Linguistics and Chinese Language Processing 8(2), 77–92 (2003)
Zhang, J., et al.: Extraction of Chinese Compound words: An experiment study on a very large corpus. In: Proceedings of the second Chinese Language Processing Workshop, HongKong, pp. 132–139 (2000)
Church, K.W., William, A.G.: Concordances for parallel text. In: Proceedings of the seventh Annual Conference of the UW Center for the New OED and Text research, Oxford, pp. 40–62 (1991)
Christopher, D.M., Hinrich, S.: Foundations of Statistical natural language processing, pp. 178–183. MIT Press, Cambridge (2001)
Feng, H.D., et al.: Accessor Variety Criteria for Chinese Word Extraction. Computational Linguistics 30(1), 75–93 (2004)
Zhang, W., Tang, X.J., Yoshida, T.: Web text mining on A Scientific Forum. International Journal of Knowledge and System Sciences 3(4), 51–59 (2006)
Zhang, W., Tang, X.J., Yoshida, T.: Text classification toward a Scientific Forum. Journal of Systems Science and Systems Engineering 16(3), 356–369 (2007)
Daille, B., et al.: Towards Automatic Extraction of Monolingual and Bilingual Terminology. In: Proceedings of the International Conference on Computational Linguistics, Kyoto, Japan, August 1994, pp. 93–98 (1994)
Fahmi, I.: C Value Method for Multi-word Term Extraction. Seminar in Statistics and Methodology. Alfa-informatica, RuG, May 23 (2005), http://odur.let.rug.nl/fahmi/talks/statistics-c-value.pdf
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhang, W., Yoshida, T., Tang, X. (2008). A Study on Multi-word Extraction from Chinese Documents. In: Ishikawa, Y., et al. Advanced Web and Network Technologies, and Applications. APWeb 2008. Lecture Notes in Computer Science, vol 4977. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-89376-9_4
Download citation
DOI: https://doi.org/10.1007/978-3-540-89376-9_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-89375-2
Online ISBN: 978-3-540-89376-9
eBook Packages: Computer ScienceComputer Science (R0)