An effective High Recall Retrieval method

Abstract

The High Recall Retrieval (HRR) problem is one of the fundamental tasks for many applications such as patent retrieval, legal search, medical search, marketing research, charging and collecting tax, and literature review, etc. Given the data set obtained by the user’s query, the HRR problem is defined as finding the full set of relevant documents while less review effort will be required. It is very expensive to review a lot of documents since most of the reviewers are experts in the specific fields such as patent attorneys, lawyers, marketing, and medical professionals. However, the existing HRR methods have been far from satisfactory to make them enumerate all relevant documents. This is due to the fact that not only the sheer volume of documents inevitably including noises (non-relevant documents) but also the threshold measurements have been inadequately adopted. To deal with these problems, we propose a novel solution to efficiently find all the relevant documents among a large set of results. It consists of two steps: (a) to effectively classify the entire documents and (b) to select the representative documents in each class. We formalized the problem and theoretically verified the upper-bound of our method. In the experiments, our method is more efficient than the state-of-the-art query expansion methods.

Introduction

High precision in traditional information retrieval systems has been important to find the most relevant targets, and it may be acceptable even if some best ones will be omitted. Recall in Patent IR (PaIR), however, is a critical issue, since the prior-art search should have been conducted without exceptions before making a full-scale business investment decision. Because a patent which is missed in the searching procedure and the infringement of the missing one might cause an enormous risk usually including a huge settlement cost, the production indemnification, and the amount reimbursed regarding the degraded Corporate Identity value.

The High Recall Retrieval (HRR) problem is one of the fundamental tasks for many applications such as patent retrieval, legal searches, medical searches, marketing research, charging and collecting taxes, and literature review, etc. These can be exemplified by situations such as when a patent examiner needs to identify all relevant patents; a lawyer needs to find every piece of evidence related to his/her case from documents that are under a legal hold; a scientist does not want to miss any piece of prior work related to his/her ongoing research; the National Tax Service imposes duties on all taxpayers exclusively.

The conventional information retrieval systems have not satisfied HRR problem, and their main purpose of them has long been focused on to maximize the precision. In HRR problem, to reduce the review efforts for the users and also not to miss any relevant results, the HRR experts have inevitably been conducted the query expression techniques which means that queries have consisted of many keywords that have repeated a lot of reformulation steps. This process has been required tedious efforts by the domain experts because the retrieval quality has been based on their ability to skillfully construct variations of queries and after that to laboriously investigate the retrieved documents. Unfortunately, the larger the collection of a target database, the more cumbersome the investigation efforts and the lower the resulting quality.

Basically, the keyword based approaches by the traditional IR methodologies have been not made successful to address the patent databases. The main reasons can be summarized as follows: (1) New keywords have been coined from each and every patent since the patent per se will be invented by a new idea or new technology. (2) The patents have been composed to hide their core keywords intentionally since they might not have been retrieved by the keyword search engine as much as possible. And (3) similar to the second reason, the patents have been tried to hide important concepts by utilizing ‘common keywords’, which will be shuffled with a lot of noise keywords. Thus, the traditional keyword-based IR technologies, as well as the cutting edge ones, have been far from the patent retrieval requirements.

As an example, consider Fig. 1 (nodes only, ignore the links that will be discussed in the next paragraph again) with a total of six documents. Assume that node 6 is a relevant document. The conventional Relevance Feedback approach determined by Rocchio method [1] will decide that node 1, 2, and 3 are individually turned out to be ”not relevant”. Accordingly, the IR system will try to find the farthest node (excluding the relevant documents) from the average values of nodes 1, 2, and 3 (0.67, 0.67, 0.67) so that the node 4 will be selected and it also is not relevant, so that the process will be repeated and no relevant document is found, until all the node has to be enumerated. This means that the Rocchio based IR system will check all documents independently and will not be able to find the right solution properly since the relevance feedback results will return an inexact threshold. Basically, this phenomenon comes from the non-relevant document decision with parameter $\gamma$ in the Rocchio formula.

However, the method we propose is capable of finding relevant documents as quickly as possible which focuses on the central nodes (representative node) and a distance threshold. Now, consider Fig. 1 as a graph model (nodes and links together) and assume that the distance are calculated using the Edit distance. (Our method is not restricted by a certain distance metric.) Suppose that the distance threshold $\epsilon =0.5$, node 5 is the central node, and the Edit distance between random nodes such as $d(1,4)$, $d(1,5)$, $d(5,6)$ are 0.33, 0.5, 0.1, respectively ($d(1,4)=(|0-0|+|0-1|+|1-1|)∕3=0.33$, $d(1,5)=(|0-0.5|+|1-0.5|+|1-0.5|)∕3=0.5$, $d(5,6)=0.1$). Based on the given threshold and the computed distances, we can find out that node 1 and 6 are not identifiable from each other since the distance between them does not satisfy the distance threshold, $d(1,6)=0.6>0.5$. Thus, the process of finding node 6 (relevant document) is performed without checking all documents independently, since it is identifiable from the central node (node 5).

Our approach is the first one for the effective examination of HRR problem with respect to 100% recall. The existing research has mostly focused on increasing recall value, but they have not been exploited to the HRR problem nor tacked to reduce the examination costs. In this study, however, we try to solve the HRR problem and to verify our method with the examination costs, that is the metric that how quickly the all relevant documents are detected.

The HRR problem with the supervised learning has been addressed to separate the relevant documents from non-relevant documents which bisect the whole document hyperplane. In supervised learning, each of the pre-selected set of documents (the “training set”) is labeled as relevant and is trained by a machine-learning algorithm, which then classifies or ranks the documents in a corpus (the “test set”) according to their likelihood of relevance. The fundamental limitation of the supervised learning approach [2] is that is valid only in the binary case of “relevant or not” so that multi-topics of patents cannot be covered. Multiple topics and multiple categories are a prerequisite for real patents since almost all patents are relevant to multiple particular IPC (International Patent Classification) codes. Another weakness of the method is For high-recall tasks, the opinion of an expert has been provided, giving their personal decision of relevance. But, obtaining authoritative opinions for even a small set of documents may have a fluctuation, or may incur unacceptable costs and time.

To the best of our knowledge, this kind of research is unprecedented. The nearest work to our study is ReQ–ReC (ReQuery–ReClassify) [3]. This research considered a scenario where a searcher requires both high precision and high recall from an interactive retrieval process. When accessing to the entire data set, an active learning loop was used to ask for additional relevance feedback labels to refine the classifier. This model uses a representational method of relevance feedback, as Rocchio [1], and machine learning method, as Support Vector Machine (SVM). The method has a restriction in the kernel function for multi-dimensions since the HRR targets such as patent documents are definitely related to multiple classifications. The ReQ–ReC, however, is only valid for a single classification that is not realistic.

Contributions To overcome the mentioned difficulties, we propose a dynamic effective method for HRR problem where our main contributions are summarized as follows:

• We develop an effective dynamic retrieval technique for HRR problem.

• We formalize a diverse retrieval method based on graph theoretic approach.

• We provide an efficient algorithm to remove the documents that cannot be among $k$ relevant ones, which can minimize the reviewing time.

• The benefits of the above features are verified through conducting experiments using various datasets.

Organizations: The rest of this paper is organized as follows. An overview of High Recall Retrieval framework and its key components is given in Section 2. High Recall Retrieval with a Single-Step (HRR ${}^1$) and Double-Step (HRR ${}^2$) describe in Sections 3 High Recall Retrieval with a single-step (, 4 High Recall Retrieval with a double-step (, respectively. Section 5 simply describes the evaluation metrics for high recall retrieval. Moreover, experimental studies are given in Section 6. We discuss related work in Section 7. Finally, we conclude the paper in Section 8.