Combining data-driven systems for improving Named Entity Recognition

Abstract

The increasing flow of digital information requires the extraction, filtering and classification of pertinent information from large volumes of texts. All these tasks greatly benefit from involving a Named Entity Recognizer (NER) in the preprocessing stage. This paper proposes a completely automatic NER system. The NER task involves not only the identification of proper names (Named Entities) in natural language text, but also their classification into a set of predefined categories, such as names of persons, organizations (companies, government organizations, committees, etc.), locations (cities, countries, rivers, etc.) and miscellaneous (movie titles, sport events, etc.). Throughout the paper, we examine the differences between language models learned by different data-driven classifiers confronted with the same NLP task, as well as ways to exploit these differences to yield a higher accuracy than the best individual classifier. Three machine learning classifiers (Hidden Markov Model, Maximum Entropy and Memory Based Learning) are trained on the same corpus in order to resolve the NE task. After comparison, their output is combined using voting strategies. A comprehensive study and experimental work on the evaluation of our system, as well as a comparison with other systems has been carried out within the framework of two specialized scientific competitions for NER, CoNLL-2002 and HAREM-2005. Finally, this paper describes the integration of our NER system in different NLP applications, in concrete Geographic Information Retrieval and Conceptual Modelling.

Introduction

The vision of the information society as a global digital community is fast becoming a reality. Progress is being driven by innovation in business and technology, and the convergence of computing, telecommunications and information systems. Access to knowledge resources in the information society is vital to both our professional and personal development. However, access alone is not enough. We need to be able to select, classify, assimilate, retrieve, filter and exploit this information, in order to enrich our collective and individual knowledge and skills. This is a key area of application for language technologies. The approach taken in this area is to develop advanced applications characterized by more intuitive natural language interfaces and content-based information analysis, extraction and filtering. Natural Language Processing (NLP) is crucial in solving these tasks. In concrete, Named Entity Recognition (NER) has emerged as an important preprocessing tool for many NLP applications such as Information Extraction (IE), Information Retrieval (IR) among other text processing applications, mainly because Name Entities (NEs) provide important cues for identifying relevant information in text.

One example of application that benefits greatly from NER is Question Answering, as, in order to answer questions like “Who is the president of the Spanish Government?”, it is useful to know that the expected answer is the name of a person, and, obviously, to consider as candidate answers only entities of this type. Accurate NE detection can also help IE systems to gain knowledge about where an event happened, who was involved in it, etc. Developing and populating ontologies is another task to be largely improved by NER. Even in automatic semantic role labelling, the extra information about an item being a person or an artifact could be crucial to determine what entity denominates the agent, depending on the type of the verb. In the area of Conceptual Modelling, a NER module is used to extract functional requirements from user’s documents written in natural language.

As can be seen, NER has crucial importance for the performance of various NLP applications. To gain a better insight into this area, we will further present a short history, together with different approaches adopted in the identification and classification of NEs.

The NER task has been first introduced by the sixth Message Understanding Conference (MUC-6). The MUC Conferences [1] have been established in 1987 with the aim of evaluating Information Extraction Systems. During MUC-6, a definition and description of the term Named Entity (NE) was introduced, together with measures for the accuracy of a system performing NER. NER involves processing a text and identifying certain occurrences of words or expressions as belonging to a particular category of Named Entities (NEs) such as names of persons, organizations, locations and numeric expressions like money or percent expressions. In fact, NER has two main sub-problems. One is NE detection (NED), that is the identification of the portion of text that forms a NE, (for example, “El Presidente Rodriguez Zapatero”¹) and the second is NE classification (NEC), the process of assigning a category label to the identified span of text (the system would assign the tag person to the NE “El Presidente Rodriguez Zapatero”).

Conferences such as MUC, CoNLL² or ACE³ proposed a limited set of categories (for example, person, organization, location, and numeric expressions like money or percent expressions), but there are many more possible labels ([2] proposed an extended Named Entity hierarchy which contains about 150 NE types). So far the research community does not commonly agree on the relationship between the number of classes and the domain of application.

Normally, an automatic NE tagger follows one of two possible approaches: the approach that employs dictionaries and hand-made rules (knowledge-based systems like the one presented in Arevalo et al. [3]) or the approach based on supervised learning techniques. The difficulty of building accurate systems following the first approach has lead to many researchers showing recently increased interest in supervised learning techniques. Supervised learning has the advantage to acquire automatically linguistic knowledge from a pool of correctly hand-annotated examples. Therefore, this type of techniques are easy to adapt to another domain or languages, but they depend on appropriate corpus. This is the main limitation of these approaches, because annotated data is not usually available. Supervised learning techniques employed so far in NER include: Decision Tree [4], Hidden Markov Model [5], [6], Maximum Entropy Model [7], Support Vector Machine [8], Boosting and voted perceptron [9], AdaBoost [10], Conditional Random Fields [11] and Finite State Automata [12].

In order to detect and classify the NEs, typically these NER systems use the surrounding context or syntactically annotated text. But it is also usual to use external resources such as lists of trigger words⁴ or gazetteers.⁵ In general, the larger the set of defined features is, the more information a classifier possesses to perform its task. Some problems that may arise are related to domain or language dependency, or over-fitting.

In fact, NER suffers the same limitations as many other NLP tasks, the bottleneck of knowledge acquisition both for corpus-based and knowledge-based systems. Knowledge-based systems rely on manually defined rules and dictionaries. This poses the difficulty of being successfully tuned to new domains or applications. On the other hand, corpus-based approaches are dependent on the availability of properly annotated corpora even when a restricted domain is needed. In both cases the high cost of acquiring such knowledge (explicit or in the form of sets of examples) is a serious handicap. Several attempts have been made to decrease the effort in acquisition and application of knowledge for NER.

The Muse system [13] is a multi-purpose NER system which focuses on this matter. One of its properties is the ability to deal with heterogeneous sources of text. [14] proposed to use active learning techniques in order to minimize the human annotation effort by automatically selecting the most useful examples for training. For the NE classification task, [15] presented a pair of algorithms based on co-training [16] that use seven simple “seed” rules. This approach is based on natural redundancy in data because, they say, both spelling and the context in which the name appears is sufficient to determine its type. Three classes of entities are location, person and organization. Other approaches based on the usage of unlabeled data were presented by [17], [18]. They applied self-training techniques to resolve the NE detection task and proposed a new voted co-training method for classify the NEs. Compared to [15] this approach did not need a split of features, but rather different machine learning classification methods that are combined through voting. The NER has been done for Spanish and the identified NEs have been classified into location, organization, person and miscellaneous categories.

Also by means of a bootstrapping technique, [19] studied the role of syntax-rich features such as constituency and dependency in order to iteratively feed a semi-supervised system based on Expectation Maximization(EM) with labeled and unlabeled data. Other approaches combine supervised and unsupervised methods with multilinguality such as [20] did for Catalan and Spanish using both AdaBoost.MH and Greedy Agreement Algorithm [21]. This system takes advantage of the syntactic similarity between the two languages. Their conclusion is that, by using multilingual resources, one can clearly outperform other approaches.

A common approach for NER and other NLP tasks, is related to the combination of several methods and classifiers that improve the performance. For example, [22] solved NER using a combination of four classifiers: robust linear classification, Maximum Entropy, transformation-based learning and Hidden Markov Model. [23] presented a system based on stacking and voting of strong classifiers, such as Support Vector Machine, boosting and memory-based learning.

Our system has been developed following this last approach. We combined several machine learning methods and linguistic knowledge resources. The main goals of the work presented in this paper are:

• to propose a completely automatic NER which involves the identification of proper names in texts, and their classification into a set of predefined categories;

• to examine the differences between language models learned by different data-driven systems performing the same NLP tasks and how they can be exploited to yield a higher accuracy than the best individual system;

• to use voting strategy to combine effectively strong classifiers such as Hidden Markov Models, Maximum Entropy and Memory-based;

• to evaluate the proposed NER system in two specialized scientific competitions for NER, such as CoNLL-2002 and HAREM-2005;

• to utilize our NER system in a NLP application such as IR;

• to evaluate the applicability of our system in the area of Conceptual Modelling.

The organization of the paper is the following: the system is described in Section 2, the conducted experiments and a discussion of the obtained results follow in Sections 3, Section 4 describes the combination of the classifiers, Section 5 presents a comparison with other NER systems, followed by the integration of the system within an IR application and a Conceptual Modelling application (Section 6). Finally, we conclude in Section 7 with a summary of the most important achievements and plans for future work.