Applying genetic algorithms to query optimization in document retrieval

Abstract

This paper proposes a novel approach to automatically retrieve keywords and then uses genetic algorithms to adapt the keyword weights. One of the contributions of the paper is to combine the Bigram (Chen, A., He, J., Xu, L., Gey, F. C., & Meggs, J. 1997. Chinese text retrieval without using a dictionary, ACM SIGIR’97, Philadelphia, PA, USA, pp. 42–49; Yang, Y.-Y., Chang, J.-S., & Chen, K.-J. 1993), Document automatic classification and ranking, Master thesis, Department of Computer Science, National Tsing Hua University) model and PAT-tree structure (Chien, L.-F., Huang, T.-I., & Chien, M.-C. 1997 Pat-tree-based keyword extraction for Chinese information retrieval, ACM SIGIR’97, Philadelphia, PA, US, pp. 50–59) to retrieve keywords. The approach extracts bigrams from documents and uses the bigrams to construct a PAT-tree to retrieve keywords. The proposed approach can retrieve any type of keywords such as technical keywords and a person’s name. Effectiveness of the proposed approach is demonstrated by comparing how effective are the keywords found by both this approach and the PAT-tree based approach. This comparison reveals that our keyword retrieval approach is as accurate as the PAT-tree based approach, yet our approach is faster and uses less memory. The study then applies genetic algorithms to tune the weight of retrieved keywords. Moreover, several documents obtained from web sites are tested and experimental results are compared with those of other approaches, indicating that the proposed approach is highly promising for applications.

Introduction

Recently, people have started dealing with an increasing number of electronic documents in information networks. Finding the documents that users need from among all the available documents is an important issue. An approach to solving this problem is to categorize documents, as in a library where, the same class of books are shelved on their own bookcase. Traditionally, document categorization has been done by humans. However, there are problems with this, in that different people may categorize the same documents differently, and people working today may produce different results to people working tomorrow. The most natural solution to this problem today is to use a computer to help people categorize documents consistently, and thus be able to retrieve items of interest easily.

Many retrieval methods assume that a query and a document are related only if they contain shared words. Documents in the same class that have the same keywords are called a “surface-based match” (Yang & Chute, 1994). Yang and Chute (1994) presented a method of document retrieval based on the idea of surface-based match. Their method learns the keyword-category association from documents that are already categorized and uses a Linear Least Squares Fit (LLSF) technique to estimate the keyword-category association. The algorithm has a time complexity of O(m²n), where m denotes the number of pairs in the training set and n represents the number of distinct words in the source space (Yang & Chute, 1994). The LLSF algorithm uses a lot of matrix operations and therefore the amount of calculation is large.

The document retrieval approach in Liddy, Paik and Yu (1994) attempts to parse the contents of a document. This method uses the “Subject Field Codes” (SFCs) from Longman’s Dictionary of Contemporary English. Because SFC maintains a lot of keywords from the Longman’s Dictionary which are already classified, it easily categorizes documents into appropriate classes by parsing the contents of documents. However, this approach is limited in that the keywords must be classified by humans, which is a difficult task.

Most document retrieval systems use keywords to retrieve documents. The systems first extract keywords from documents and then assign weights to the keywords by using different approaches. Such a system has two major problems. One is how to extract keywords precisely (Baeza-Yates, 1992; Chen, He, Xu, Gey & Meggs, 1997; Chien, Huang & Chien, 1997; He et al., 1996, Kwok, 1997; Nie, Brisebois & Ren, 1996; Zhai, Tong, Milic-Frayling & Evans, 1996) and the other is how to decide the weight of each keyword (Gordon, 1988, Lewis et al., 1996). We try to solve the two problems in this paper.

Retrieving keywords in Chinese documents is especially difficult since Chinese sentences lack explicit word boundaries. Chien et al. (1997) used an automatic statistics-based approach which efficiently extracts significant lexical patterns from a set of relevant documents. This approach used a data structure called PAT tree (Gonnet, Baeza-Yates & Snider, 1992) to index full-text documents. The approach in Chien et al. (1997) inserted all possible strings of documents into a PAT tree, making the PAT tree a fast and easy means of searching for words. However, retrieving keywords requires much time.

Nie et al. (1996) proposed a hybrid segmentation approach which combines several commonly used approaches, including statistical and dictionary approaches. In the hybrid approach, each word for which statistical information is available, is used in priority, while the others are stored in the dictionary and then assigned a default probability. The highest product of the various words’ probabilities is the best solution of the segmented strings (Nie et al., 1996). The hybrid approach finds exact keywords by the dictionary approach, and extracts new keywords by the statistical approach. To be put into practice, however, the approach needs information such as a dictionary and a set of heuristic rules.

As for deciding of the weight of each term, the simplest way is to make the weight the frequency with which the term occurs (TF) in the documents. If there is a large amount of documents, the terms would occur frequently. Thus, one normalization typically used in weighting algorithms compensates for the number of words an item has. Buckley, Singhal and Mitra (1995) presented the term frequency weighting formula as follows:

$$\text{(1+}\text{log}\text{(}\text{TF}\text{))/(1+}\text{log}\text{(}\text{average}\text{(}\text{TF}\text{))}\text{(1−}\text{slope}\text{)}{}^{\mathrm{\ast }}\text{pivot}\text{+}\text{slope}{}^{\mathrm{\ast }}\text{number-of-unique-term}$$

where slope was set to 0.2 and pivot was set to the average number of each term occurring in the documents. Jones (1972) presented the following Inverse Document Frequency (IDF) measure:

$$\text{IDF}{}_{\mathrm{i}}\text{=}\text{log}{}_2\text{N}\text{n}{}_{\mathrm{i}}\text{+1,}$$

where N is the number of documents and n_i is the total number of documents containing the term i. Several methods are presented to combine TF with the IDF measure. The approach proposed by Salton and Buckley (1988) is given below:

$$\text{w}{}_{\mathrm{ij}}\text{=}\text{0.5+}\text{0.5}{}^{\mathrm{\ast }}\text{freq}{}_{\mathrm{j}}\text{maxfreq}{}_{\mathrm{j}}{}^{\mathrm{\ast }}\text{ID}{}_{\mathrm{i}}$$

where freq_ij is the frequency of term i in the document j and maxfreq_j is the maximum frequency of any term in the document j. Lochbaum and Streeter (1989) presented the following entropy measure in several experiments:

$$\text{entropy}{}_{\mathrm{i}}\text{=1−}\text{∑}\text{k=1}\text{N}\text{Freq}{}_{\mathrm{ik}}\text{TFreq}{}_{\mathrm{i}}\text{log}{}_2\text{TFreq}{}_{\mathrm{i}}\text{Freq}{}_{\mathrm{ik}}\text{log}{}_2\text{N}\text{,}$$

where N is the number of documents; Freq_ik is the frequency of term i in the document k; TFreq_i is the total frequency of term i.

The above measures are used in traditional IR. Approaches such as TF and IDF compute the weights of certain of terms. Yang and Korfhage (1993) were the first to use genetic algorithms for query optimization in information retrieval. Their work emphasizes only term weight modification, and does not expand queries. This paper presents an alternative approach to finding keywords of documents and then applies a genetic approach to adapt the weights of keywords.

The remainder of the paper is organized as follows. In Section 2, we present our system framework. In Section 3, we combine the Bigram model (Chen et al., 1997; Yang, Chang & Chen, 1993) and the PAT-tree based filter algorithm (Chien et al., 1997) to retrieve keywords. In Section 4, we present a training algorithm using genetic algorithms (Chang and Hsu, 1997, Chang and Hsu, 1999, Goldberg, 1989, Yang and Korfhage, 1993) to adapt the weights of keywords. In Section 5, a relevance feedback mechanism is presented. In Section 6, several kinds of Chinese documents are designed to test the performance of our approach. Conclusions are drawn in Section 8.