A Memory-Based Learning Approach for Named Entity Recognition in Hindi

1 Introduction

Named entity (NE) recognition (NER) is an important preprocessing component of almost all natural language processing (NLP) applications, such as information extraction, machine translation [1], question answering [16], text summarization, information retrieval, ontology development (e.g. clinical vocabularies may be developed extracting directly from clinical reports [15]).

The objective of NER is to identify NEs in a running text and map them to predefined classes such as location names (cities, countries, places, etc.), person names (names of people), organization names (companies, government organizations, committees, etc.), temporal expression (date, time, period), miscellaneous names (monetary expression, materials, artifacts, quantity, measurement expressions, etc.), and “none of the above.” NER in an Indian language like Hindi is a challenging task because, unlike the English language, Indian languages lack the capitalization information that plays a very important role to identify NEs.

The term NER task was first coined in the 6th and 7th Message Understanding Conference. This conference was called for discussing the problem of recognition of names, temporal expression, and monetary expression in documents. It was later defined as the shared task of the CoNLL 2003 conference, where the task was to tag noun phrases with four classes: person (PER), organization (ORG), location (LOC), and miscellaneous (MISC).

In this paper, we present a memory-based learner for NER in Hindi. Hindi is the third most spoken language in the world, and it is the national language of India. The work reported in this paper is different from existing NER systems in terms of the following points:

• In many previous works on NER for Hindi [8, 17, 19], the researchers only considered four broad NE categories, namely person names, location names, organization names, and miscellaneous, whereas we have considered a more fine-grained set of 22 NE categories that can be grouped into three broad types: (i) NE categories for name expressions – person, organization, location, facilities, locomotive, artifact, entertainment, materials, living things, plants, disease; (ii) NE categories for number expressions – distance, money, quantity, count; (iii) NE categories for time expressions – time, year, month, date, day, period, Sday (special day such as Christmas day).

• We have used a set of composite features along with a set of individual features for NER.

• We have employed a memory-based learning method for the Hindi NER task.

2 Related Work

The NER task has received more attention from NLP researchers since the last decades due to the importance of NER in many NLP tasks: (i) information extraction that seeks to locate and classify atomic elements in the text; (ii) question answering, which also requires extracting important information from questions and generating appropriate answers; (iii) machine translation; and (iv) other applications. Three commonly used approaches to NER are the linguistic approach, machine learning (ML)-based approach, and hybrid approach.

The linguistic approach [4] mainly uses handcrafted rules that are created by linguists. Clearly, the skills and experience of a linguist are more important to reach the desired performance by the overall system. The main advantage in the rule-based approach is extraction of complex entities can be fine tuned, and it does not require a large amount of annotated data or corpus. Handcrafted rules are language dependent, and so they may not be applicable for other languages.

The ML-based technique for NER tasks requires large amounts of NE-annotated training data to acquire a higher level of language knowledge. Sekine [23] used decision tree learning for Japanese NE recognition. The most commonly used ML methods for the NER task are the hidden Markov model (HMM), conditional random fields (CRFs), and support vector machine (SVM). Each of these ML approaches has its own advantages and disadvantages. HMM [2] was found to be very effective in sequential labeling problems. CRF [9] is a probabilistic approach, flexible to capture many closely connected features, including overlapping and non-independent features. SVM [24, 25] separates the classes by drawing a decision boundary between them, and a query instance is predicted to belong to a category based on which side of the gap it falls on.

Saha et al. [18] presented a hybrid approach that combines rule-based and ML-based methods, and created new methods that take the strongest points from each method. They also employed the Gazetteer Lists in NER tasks. They experimented with Hindi NER using a training set of >5 lakh words and a test set of 38,704 words, and achieved F-measures of 66.08, 67.50, and 65.13, respectively, for maximal, nested, and lexical-level evaluation.

Cucerzan and Yarowsky [6] studied the NER task for Hindi as a part of their language-independent NER work that used morphological and contextual evidences [6]. They experimented with five languages – Romanian, English, Greek, Turkish, and Hindi. Among these, the accuracy for Hindi was the worst. For the Hindi language, they achieved a 41.70% F-measure with a very low recall of 27.84% and about 85% precision. Li and McCallum [14] presented a more successful Hindi NER system that uses CRFs with feature induction. They were able to achieve a 71.50% F-value using a training set of size 340k words. For Hindi, better accuracy was achieved by Kumar and Bhattacharyya [13]. Their maximum entropy Markov model-based system gives a 79.7% F-value.

The k-nearest neighbor (KNN) method that we have used for the NER task is a memory-based learning method [20, 22, 27]. Compared to the previous ML-based approach, the main advantage of a memory-based NER system is that it is less affected by the sparse data problem, as the KNN approach provides a solution to the sparse data problem via an implicit similarity-based smoothing scheme. Our choice of the KNN approach was motivated by its simplicity, flexibility to incorporate different data types, adaptability to irregular feature spaces, and capability to directly handle string features that facilitate defining the context of words.

3 Data Set Description

The data set that we have used in our experiment has been taken from NLP tool contest on NER for Indian Languages, conducted in association with ICON 2013 (http://ltrc.iiit.ac.in/icon/2013/nlptools/). The data sets released for the tool contest was POS tagged and chunked. The goal of this contest was to perform NE recognition on a variety of types such as artifact, entertainment, facilities, location, locomotive, materials, organisms, organization, person, plants, count, distance, money, quantity, date, day, period, time, and year.

The data set is available in SSF (Shakti Standard Format) (http://ltrc.iiit.ac.in/nlptools2010/files/documents/SSF.pdf). Sentence-level SSF is used to store the analysis of a sentence. The analysis of the sentence gives POS and chunk information for the tokens in the sentence. Each line represents a token or a group information (except for lines with “)),” which only indicates the end of a group). For each group, the symbol used to indicate the start of a group is “((.” Each token or group information has three parts: the first part stores the tree address of each token or group (this is for human readability only); the second part stores the token or group information; and the third part stores the POS tag or group/phrase category (chunk information).

To enhance the performance of our proposed model, we have considered chunk boundaries as the tokens in our model. The chunk boundaries are indicated by double opening brackets “((” and double closing brackets “)).” This is because, most of the time, these chunk boundaries play a role as a separator for the NEs, and this can be helpful to find the NE boundaries. For example, consider the sample sentence in Figure 1 containing NEs. As we can see in Figure 1, chunk boundaries (opening and closing double brackets) play a separator role for NE recognition. Here, from the chunk boundaries, we can have partial knowledge of NE boundaries within the sentence.

Figure 1:

A Sample Training Sentence in SSF Format.

In the ICON 2013 NLP tool contest on NER for Indian Languages, three labeled data sets were initially given to contestants to evaluate their systems: training set, development set, and test set. The detailed description on our used experimental data sets is given in Table 1.

4 Memory-Based Learning Approach

We have used the memory-based learning method for Hindi NER, although a limited number of works on NER has used memory-based learning for NER. The memory-based learning algorithm that we have used for the NER task is the KNN algorithm [10], which works as follows:

• Stores the training data into memory.

• When a query instance is encountered, k numbers of nearest neighbors of the query are retrieved from the memory by computing the similarities between the training instances and query instance.

• Finally, based on class distribution of these k-retrieved instances, the query instance is classified. The KNN approximates the class label of query instance by assigning the most frequent class label occurring among the k-most similar patterns retrieved from the memory.

The steps of the KNN algorithm are elaborated in Figure 2.

Figure 2:

KNN Algorithm.

5 Feature Set

Features play a crucial role in identifying and classifying the NEs. While assigning NE tags to the tokens in a sentence, it is scanned from the left to the right and the KNN classifier is used to assign the tags to the tokens one by one. Here, a token is represented as a feature vector that is a vector of values of the features describing the token. In our settings, a training instance is a feature vector representing a token labeled with a particular NE tag and a test instance is an unlabeled feature vector representing a token to be labeled. For this task, we have considered a feature set that includes a variety of simple features and their combinations. We have divided our entire features into two sets: single or simple features and composite features.

5.2 Composite Features

We have used several combinations of features to act as a single feature, i.e. combining the effect of several features as a single feature. As all NEs are noun phrases, and it is quite difficult to classify them into different entity classes, we have observed that these noun entities are classified with the help of several composite features that are nothing but the combination of several single features. The main objective of using these features is to correctly classify tokens by using their associated surrounding information. For example, consider the following two sentences.

1. England won the World Cup (England ne World Cup jeeta).

2. The World Cup took place in England (World Cup England mein khela gaya).

Here, in sentence 1, the word England is an Organization name and, in sentence 2, it is a Location name. For resolving this ambiguity, we can associate several contextual features with the current token (England) to generate a single feature that is expected to have the ability to resolve ambiguity to some extent.

As the objective of having the composite features is to utilize contextual information in resolving tag ambiguities, we have considered short context and relatively long context both while designing composite features. This leads to a number of composite features that are finally optimized by using the back elimination method. The different composite features that we have initially considered are discussed below. To illustrate the values of the composite features, we take an example of a tagged sentence shown in Figure 3.

Figure 3:

A Sample POS-Tagged and Chunked Training Sentence (Chosen from the Training Data).

5.2.1 Current Token and POS Tag of the Next Token and NE Tag of the Previous Token

This feature is a combination of the current token, POS tag of the next token, and NE tag of the previous token. If we consider the above sequence as a composite feature, then the attribute values for different single features will be taken together. For example, for the first word in the sentence shown in Figure 3, this composite feature has the following parts:

1. October.

2. END [the tag “END” is not part of the IOB format; it is inserted by us to assign a tag to the token “))”].

3. “O” (outside) (because the current token is the first word of the sentence).

If the current token is an entity, then this feature may help determine whether the current token is a single-word NE or if it is a part of a multiword NE. For example, if the current token is a single-word entity, it should be tagged as XXX-B; if it is middle part of an entity, it should be tagged as XXX-I, where XXX is an entity name like Person, Location, etc.

When comparing two instances on this feature, the exact match (all parts match) is considered. The similarity value is set to 1 for an exact match, 0 otherwise. In other words, the value of the similarity between two instances under comparison is incremented by 1 when an exact match between the values corresponding to this feature component is found; otherwise, no increment is made.

If nothing is specified, for the remaining composite features discussed below, the same rule is also applied while computing the similarity between any two instances.

5.2.2 Current Token and Previous Token’s Chunk Information and NE Tag of the Previous Token

This feature is a combination of the current token, the previous token’s chunk information, and the NE tag of the previous token. For example, for the first word in the sentence shown in Figure 3, this composite feature has the following parts:

1. October.

2. NP-B [previous token “((” has the chunk tag NP-B as per the IOB format].

3. O.

When comparing two instances on this feature component, the exact match (all parts match) is considered. The similarity value is set to 1 for an exact match, otherwise 0.

This feature is designed to target at detecting whether the current token is the beginning of an NE or not.

5.2.3 Current Token and POS Tag of the Next Token and NE Tag of the Previous Token and Chunk Information of the Next Token

This feature is a combination of the current token, POS tag of the next token, NE tag of the previous token, and chunk information of the next token. For example, for the first word in the sentence shown in Figure 3, the composite feature has the following parts:

1. October.

2. END.

3. O.

4. NP-E [this is also assigned by us for the token “))”].

When comparing two instances on this feature component, the exact match is considered. The component similarity value is set to 1 for a match, otherwise 0.

This feature is designed to target at detecting whether the current token is at the end of an NE or not.

5.2.4 Current Token and POS Tag of the Next Token and NE Tag of the Previous Token and Chunk Information of the Next Token and POS Tag of the Next of Next Token

This feature is a combination of several single features that include the current token, POS tag of the next token, NE tag of the previous token, chunk information of the next token, and POS tag of the next of next token. For example, for the first word in the sentence shown in Figure 3, the composite feature has the following parts:

1. October.

2. END.

3. O.

4. NP-E.

5. CCP-Open.

When comparing two instances on this feature component, the exact match is considered. The component similarity value is set to 1 for a match, otherwise 0.

This feature is assumed to utilize the left and right context of the current token to distinguish among the different types of NEs.

5.2.5 Current Token and NE Tag of the Previous Token and Punctuation Symbols

This feature includes only three single features called current token, NE tag of the previous token, and punctuation symbols. For example, for the first word in the sentence shown in Figure 3, the composite feature has the following parts:

1. October.

2. O.

3. no (If the current token is a dot, comma, or hyphen, then its value is set to “yes,” and “no” otherwise.).

The main purpose of using this feature is to find the continuation of NE class where punctuation symbols (dot, comma, and hyphen) appeared within NE classes. For example:

• <Location> Uttar – Pradesh </Location>

• <Entertainment> River – Rafting </Entertainment>

In the above example, we can see that hyphen is present within NE class (Location and Entertainment). Similarly, it helps to find the continuation of NE within some digit patterns where digits are separated by some punctuation symbols; for example, the following sequence of tokens contains hyphenated numbers and a number containing a comma inside.

• <Period> 1479 – 1531 </Period>

• <Count> 10, 000 </Count>

When comparing two instances on this feature component, the exact match is considered. The component similarity value is set to 1 for a match, otherwise 0.

5.2.6 Current Token and Previous Token

This feature includes only two single features: current token and previous token to create a composite feature.

For example, for the first word in the sentence shown in Figure 3, the composite feature has the following parts:

1. October.

2. ((.

The value of this composite feature is the combination of the values of the current token and the previous token.

When we compare two instances on this feature component, we set the component similarity value to 1 if both have the same value for this feature component. However, if the values match partially (i.e. the value of the current token portion in the composite feature value matches, but only the suffix or the prefix of the previous token gets matched), then we set the component similarity value to a value (0.8), which is <1, to discriminate between a full match and a partial match. We set the component similarity to 0 if the current token is not matched at all.

We assume that this feature may distinguish between the types of NEs. For example, in the sentence segments “England won” and “in England,” the first occurrence of “England” is the organization name and the second occurrence is the name of a place.

5.2.7 Current Token and Next Token

This feature includes only two single features, current token and next token, to create a composite feature.

For example, for the first word in the sentence shown in Figure 3, the composite feature has the following parts:

1. October.

2. )).

When we compare two instances on this feature component, the component similarity value is computed in the same way as it is done for “current token and previous token.”

We assume that this feature may also distinguish between the types of NEs. For example, in the sentence segments “England is a place for” and “England won,” the first occurrence of “England” is a place name and the second occurrence of “England” is an organization name.

5.2.8 Current Token, Previous POS, and Previous Chunk

This feature includes only three single features, current token, previous POS, and previous chunk, to create a composite feature.

For example, for the first word in the sentence shown in Figure 3, the composite feature has the following parts:

1. October.

2. NP-OPEN.

3. NP-B [previous token “((” has the chunk tag NP-B as per the IOB format].

When comparing two instances on this feature component, the exact match is considered. The component similarity value is set to 1 for a match, otherwise 0.

Using this feature, we associate the current token with the POS tag of the previous token and chunk information of the previous token. The objective here is to utilize the left context of the current token in determining its NE tag.

5.2.9 Current Token, Next POS, and Next Chunk

This feature includes only three single features: current token, next POS, and next chunk.For example, for the first word in the sentence shown in Figure 3, the composite feature has the following parts:

1. October.

2. END.

3. NP-E.

When comparing two instances on this feature component, the exact match is considered. The component similarity value is set to 1 for a match, otherwise 0.

With this feature, we associate the current token with its right context to utilize it in determining the NE tag of the current token.

5.2.10 Current Token, Previous POS, Previous Chunk, Next POS, and Next Chunk

This feature includes only five single features: current token, previous POS, previous chunk, next POS, and next chunk.

For example, for the first word in the sentence shown in Figure 3, the composite feature has the following parts:

1. October.

2. NP-OPEN.

3. NP-B.

4. END.

5. NP-E.

When comparing two instances on this feature component, the exact match is considered. The component similarity value is set to 1 for a match, otherwise 0.

With this feature, we utilize both the left context and right context of the current token in determining the NE tag to be assigned to it.

6 Postprocessing Rules

Several postprocessing techniques have been adopted in order to increase the performance of a classifier by resolving the erroneous tags. An analysis was made on the most frequent errors made by our model, and rules were crafted to correct errors:

1. If a token is tagged as NE-I (where NE can be any NE tag such as PER, LOC, ORG, etc.) and the previous token is tagged as O (outside tag), then we replace the NE-I tag by NE-B; that is, any isolated NE-I is replaced by NE-B.

2. A second case of erroneous tagging is encountered when more than one NE tags are appeared within a tag sequence, e.g. “O PER-B PER-I LOC-I O.” One possible solution for this type of error is to assign the NE whose tags are more frequent in the sequence. However, this may not work when there are at least two NEs whose tags are occurring in equal number in the tag sequence; for example, consider the following situation:

“O PER-B LOC-I LOC-I PER-I O,” where the tags for the NEs PER (person) and LOC (location) have the same frequency, i.e. 2.

To handle this situation, we have used the confidence value of the classifier in assigning an NE tag. The total sum of the confidence values over the tags for each possible NE is computed, and the sequence is assigned the NE category for which the highest total confidence value is obtained. The confidence value of the classifier in assigning an NE tag to a token is calculated by taking the ratio of the frequency of the particular NE tag in the group of the k labels of the k nearest neighbors and the value of k.

According to the above-mentioned method, to break the tie situation for the “O PER-B LOC-I LOC-I PER-I O,” we have to sum up the confidence values for the tags PER-B and PER-I and also sum up the confidence values for the tags LOC-I and LOC-I. Finally, the two values are compared to decide whether the token sequence corresponding to the tag sequence should be classified as PERSON NE or LOCATION NE. If the token sequence is finally classified as PERSON NE, the corresponding tag sequence is converted in the IOB format before writing the token sequence and the associated tag sequence to the output file; that is, “O PER-B LOC-I LOC-I PER-I O” is converted to “O PER-B PER-I PER-I PER-I O.”

7 Backward Elimination Method

One of the problems with the KNN algorithm is that it suffers from the curse of dimensionality problem [26]. For this problem, the neighborhood of a given point becomes very sparse in a high-dimensional space, resulting in high variance. One of the solutions of this curse of dimensionality problem is to shrink the unimportant dimensions of the feature space, bringing more relevant neighbors close to the target point. One of the approaches to overcome this curse of dimensionality problem is the backward elimination approach [5]. In our work, for removing the relatively irrelevant attributes, we have used the backward elimination method, which works as follows:

• Starts initially with the full set of features and greedily removes the one that most improves performance.

Using the backward elimination method, we have identified a set of features whose removal improves the system performance. The detail of the identified unimportant features is discussed in Section 10.

8 HMM-Based NER Model

To prove the effectiveness of our proposed method, we have compared the performance of our proposed NER system with an HMM-based NER system that was submitted and evaluated in the ICON 2013 NLP tool contest (http://ltrc.iiit.ac.in/icon/2013/nlptools/). An HMM-based NE tagging approach presented in Ref. [11] considers NE tagging as a sequence labeling task. In general, like POS tagging [21], an HMM-based NE tagging task commonly uses words in a sentence as an observation sequence. However, as the data released for the ICON 2013 NLP tool contest was POS tagged and chunked, the system presented in Ref. [11] uses this additional information: POS tag and chunk tag for the NER task. POS tag and chunk tag are incorporated into the observation symbol, which then becomes a triplet <word, POS-tag, chunk-tag> and hence the observation sequence for a sentence with the word sequence <word₁, word₂, …, word_n> is considered as (<word₁, POS-tag₁, chunk-tag₁>, <word₂, POS-tag₂, chunk-tag₂>, <word₃, POS-tag₃, chunk-tag₃>, …, <word_n, POS-tag_n, chunk-tag_n>).

For an HMM-based NER model, the important issues are the data sparseness problem, decoding to find the best hidden state sequence given an input HMM, and a sequence of observations and handling of unknown observation tokens. When we implement the HMM-based NER model, we have handled these issues in the following ways.

To handle the data sparseness problem, the deleted interpolation-based smoothing technique [3] has been used. For decoding purposes, the viterbi decoding algorithm has been used. The task of a decoder is to find the best hidden state sequence given an input HMM and a sequence of observations. To handle the unknown triplets in the observation sequence, the observation probability of an unknown one is estimated by analyzing the POS tag, chunk tag, and the suffix of the word that constitute the triplet. The observation probabilities of the unknown triplet <word, POS-tag, chunk-tag> corresponding to a word in the input sentence are decided according to the suffix of a pseudo-word formed by adding the POS tag and chunk tag to the end of the word. We find the observation probabilities of such unknown pseudo-words using suffix analysis. To handle unknown observation through suffix analysis, when a list of rare words is created, the threshold on the frequency of words in the training corpus is tuned by considering the different possible threshold values. The different possible suffix lengths are also considered while calculating observation probabilities of an unknown observation based on its suffix information. The HMM system obtains the best results on the development set when we consider that the words whose frequency in the training corpus is ≤2 are the rare words and the maximum suffix length is 9. The detail of this HMM-based NE tagger can be found in Ref. [11].

The performance of this HMM-based NER system is also tested on the test data set on which our proposed memory-based NER system is tested. As we have used the Gazetteer feature in the KNN-based NER system, we should incorporate the same in the HMM-based system to make them comparable. To incorporate this additional feature in the HMM-based NER system described in Ref. [11], we deploy the Gazetteer list in the training data when the HMM-based system is trained. This actually affects the observation probability of the words that are found in the Gazetteer list.

9 Performance Measures

For system evaluation, we use “exact match evaluation” and accordingly we calculate precision, recall, and F-measure.

Precision is defined as the percentage of NEs found by the system that are correct, and recall is defined as the percentage of NEs present in the solution (answer file) that are found by the system. As our system produces output in the IOB format, “exact match” means the system-assigned labels for the parts of an NE match in order with those for the corresponding NE in the solution.

For example, if there are five true NEs in the answer file, there are five system guesses and only one guess that exactly matches the solution. The precision is therefore 20% and the recall is 20%.

The F-measure is a combination of precision and recall and is calculated as follows:

F-measure = (β2 + 1) ∗ P ∗ R(P + R),

where P and R are the precision and recall, respectively. β is a weighting between precision and recall. For system evaluation, we set the value of β to 1.

10 Results

11 Evaluation Results of the 10-Fold Cross-Validation Test

To compare the system performance, a 10-fold cross-validation test is performed for each system. For this purpose, we use the entire labeled data (combining three labeled data sets initially given to contestants: training set, development set, and test set) released for the NLP tool contest held in association with ICON 2013. In this evaluation method, the entire data is divided into equal 10 parts (each part contains the equal number of sentences) and one part is held out for testing and the remaining 9 parts are combined to use for training the system. Thus, for each system, 10 test results are collected for the 10 different folds.

A statistical analysis is carried out on the results of the 10-fold cross-validation test obtained by the HMM-based system and the best version of our proposed memory-based NER system. We tested the statistical significance of the difference between precisions of the two systems, as well as their recalls, using a paired t-test. Results are reported in Table 8.

As we can see from Table 8, the average precision of the memory-based NER system is better than the HMM-based NER system, and the difference between the precisions of the two systems is statistically significant (p<0.05), whereas the difference between the recalls of two systems is not statistically significant. In Table 8, p-value shown in italic font indicates statistical significance of the recall difference.

Compared to the HMM system, the proposed memory-based NER system shows an improvement in precision but no significant changes in recall is found. The main drawback of the HMM-based system is that it depends on the suffix of the unknown words (words that are not present in training corpus) for predicting its NE tag, and the similar suffix may also present in the words of different NE categories. The data sparseness problem is also another issue in HMM-based systems. Though we have used a smoothing technique to handle this situation, no smoothing technique is error free. On the other hand, memory-based learning methods like KNN are less affected by problems related to unknown word handling. We can easily incorporate many contextual features and word-level features in KNN-based NER systems. In our work, we have incorporated a number of contextual and word-level features in KNN. We have also incorporated the Gazetteer feature in the KNN-based system in a way that is different from that used in HMM-based systems. This helps in improving the precision of KNN.

However, we did not find any significant difference in the recall values of the two systems compared. The reason is the presence of annotation errors in the training data as well as test data. We observed the following types of error:

1. A word sequence has been annotated as NE type X, but it should be annotated as NE type Y.

2. A word sequence is not an NE, but it has been annotated as NE.

3. A word sequence is an NE, but it is not annotated as NE.

We have observed the frequent occurrence of the error type 1 in the training data. This type of annotation error occurs for the cases where NE types are very close in meaning (e.g. COUNT and QUANTITY, PERIOD and TIME, etc.). In Figure 4, we have shown some examples of annotation errors. The word sequences shown in bold font have been incorrectly annotated in the data set. We have not changed it manually because this data set was not created by us. The data set that we have used in our experiment has been taken from the NLP tool contest on NER for Indian languages, conducted in association with ICON 2013. Figure 4 shows that assignment of “QUANTITY” and “COUNT” NE tags to word sequences is not consistent.

Figure 4:

Examples of Annotation Errors.

The annotation errors affect precision and recall both. For example, assume that a sentence <w₁ w₂ w₃ … w_n> with several single-word NEs is annotated as <w₁/NE-X w₂/O w₃/NE-Y w₄/O w₅/O w₆/NE-X w₇/O W₈/NE-Z> and the annotation errors have occurred at w₁/NE-X and w₆/NE-X. Actually, the correct annotation is w₁/NE-Y and w₆/NE-Y. If the system could tag without any error, it would tag the sentence ideally as w₁/NE-Y w₂/O w₃/NE-Y w₄/O w₅/O w₆/NE-Y w₇/O W₈/NE-Z. In this case, comparing the system-provided tags and the human-provided tags (where annotation errors are not corrected) for the NEs in the sentence <w₁ w₂ w₃ … w_n>, we have precision=2/4 and recall=2/4. However, if the system behaves differently and assigns tags to the sentence as “w₁/O w₂/O w₃/NE-Y w₄/O w₅/O w₆/O w₇/O W₈/NE-Z,” then precision is 2/2 and recall is 2/4. In the second case, precision improves, but recall remains the same as before. Most likely, this is why our proposed memory-based learning approach has obtained an improved precision score without any significant changes in the recall score. Comparing these two situations, we can find that the second case is better than the first one. We should also prefer the second case because recognizing a word sequence as “not an entity” is better than recognizing it as an incorrect entity. We suggest that managing noise in the data set is an important issue related to improvement in recall as well.

12 Conclusion and Future Work

A memory-based NE recognizer for Hindi has been presented in this paper. The performance of the memory-based Hindi NE recognizer has been compared to a trigram HMM-based Hindi NE recognizer. A comprehensive set of features has been used for this task. The experimental results show that the performance of the proposed memory-based NER system is comparable to the HMM-based NER system. The NER task we have considered in our study is more challenging in a sense that our used data set considers 22 NE categories, whereas many previous studies on NER have considered mainly the three or four NE categories such as person name, location name, organization name, and miscellaneous.

One of the major drawbacks of our proposed memory-based NE recognizer is that it is slow in nature. We have planned to apply an efficient KNN search algorithm [28] for speeding up the proposed NE recognizer.