Abstract
In learning a new word by a dictionary, we first need to know a set of “basic words” which are frequently appeared in word definitions. It often happens that you cannot understand the word you looked up because there are still some words you do not understand in its definitions or explanations provided by the dictionary. You can keep looking up these new words recursively till they all can be well explained by some basic words you already knew. How to automatically find a minimum set of such basic words to define (or recursively define) the entire vocabulary in a given dictionary is what are going to discuss in this paper. We propose an efficient algorithm to construct the Minimum Vocabulary (MV) using the word frequency information. The minimum vocabulary can be used for language modeling and experimental results demonstrate the effectiveness of using the minimum vocabulary as features in text classification.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Armstrong, W.W.: Dependency structures of data base relationships. Information Processing 74, 580–583 (1974)
Cancho, R.F.I., Solé, R.V.: Least effort and the origins of scaling in human language. Proceedings of the National Academy of Science 100, 788–791 (2003)
Chandrasekharan, N., Sridhar, R., Iyengar, S.: On the minimum vocabulary problem. Journal of the American Society for Information Science 38(4), 234–238 (1987)
Gabrilovich, E., Markovitch: Text categorization with many redundant features: Using aggressive feature selection to make SVMs competitive with C4.5. In: ICML (2004)
Hazenberg, S., Hulstun, J.H.: Defining a minimal receptive second-language vocabulary for non- native university students: An empirical investigation. Applied Linguistics 17(2), 145–163 (1996)
Hirsh, D., Nation, P.: What vocabulary size is needed to read unsimplified texts for pleasure? Reading in a Foreign Language 8, 689–696 (1992)
Kilgarriff, A.: Putting frequencies in the dictionary. International Journal of Lexicography 10(2), 135–155 (1997)
Kilgarriff, A.: Using word frequency lists to measure corpus homogeneity and similarity between corpora. In: ACL-SIGDAT Workshop on Very Large Corpora, pp. 231-245 (1997)
Loper, E., Bird, S.: NLTK: The natural language toolkit. In: Proceedings of the ACL 2002 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, vol. 1, pp. 63–70 (2002)
Qin, Z., Thint, M., Huang, Z.: Ranking Answers by Hierarchical Topic Models. In: Chien, B.-C., Hong, T.-P., Chen, S.-M., Ali, M. (eds.) IEA/AIE 2009. LNCS (LNAI), vol. 5579, pp. 103–112. Springer, Heidelberg (2009)
Ramakrishnan, R., Gehrke, J.: Database Management Systems. McGraw-Hill, Inc. (1999)
Saiedian, H., Spencer, T.: An efficient algorithm to compute the candidate keys of a relational database schema. The Computer Journal 39(2), 124–132 (1996)
West, M.P.: A General Service List of English Words. Longman (1976)
Zhao, Q., Qin, Z., Wan, T.: What Is the Basic Semantic Unit of Chinese Language? A Computational Approach Based on Topic Models. In: Kanazawa, M., Kornai, A., Kracht, M., Seki, H. (eds.) MOL 12. LNCS (LNAI), vol. 6878, pp. 143–157. Springer, Heidelberg (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Lin, S., Qin, Z., Huang, Z., Wan, T. (2012). An Efficient Minimum Vocabulary Construction Algorithm for Language Modeling. In: Jiang, H., Ding, W., Ali, M., Wu, X. (eds) Advanced Research in Applied Artificial Intelligence. IEA/AIE 2012. Lecture Notes in Computer Science(), vol 7345. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31087-4_31
Download citation
DOI: https://doi.org/10.1007/978-3-642-31087-4_31
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-31086-7
Online ISBN: 978-3-642-31087-4
eBook Packages: Computer ScienceComputer Science (R0)