Corpus Linguistics for establishing the natural language content of Digital Library documents

Abstract

The methods of corpus linguistics can reveal a great deal of information about word use and language structure by careful processing of very large corpora. This information can be used for adding organizational structure to digital libraries both in terms of individual document content and inter-document relations. The structure discovered by corpus linguistics methods reflects the actual use of words and language style in particular domains and genres, rather than being constrained by pre-built categories. The data presented here has demonstrated the power of simple word classification methods for discovering semantically related word clusters. Work in progress based on the new balanced entropy principle overcomes a number of limitations of current classification methods and should discover more detailed and accurate information about word relations and text structure.