Evaluation of machine learning-based information extraction algorithms: criticisms and recommendations

Abstract

We survey the evaluation methodology adopted in information extraction (IE), as defined in a few different efforts applying machine learning (ML) to IE. We identify a number of critical issues that hamper comparison of the results obtained by different researchers. Some of these issues are common to other NLP-related tasks: e.g., the difficulty of exactly identifying the effects on performance of the data (sample selection and sample size), of the domain theory (features selected), and of algorithm parameter settings. Some issues are specific to IE: how leniently to assess inexact identification of filler boundaries, the possibility of multiple fillers for a slot, and how the counting is performed. We argue that, when specifying an IE task, these issues should be explicitly addressed, and a number of methodological characteristics should be clearly defined. To empirically verify the practical impact of the issues mentioned above, we perform a survey of the results of different algorithms when applied to a few standard datasets. The survey shows a serious lack of consensus on these issues, which makes it difficult to draw firm conclusions on a comparative evaluation of the algorithms. Our aim is to elaborate a clear and detailed experimental methodology and propose it to the IE community. Widespread agreement on this proposal should lead to future IE comparative evaluations that are fair and reliable. To demonstrate the way the methodology is to be applied we have organized and run a comparative evaluation of ML-based IE systems (the Pascal Challenge on ML-based IE) where the principles described in this article are put into practice. In this article we describe the proposed methodology and its motivations. The Pascal evaluation is then described and its results presented.

Appendix

1.1 Statistical significance testing

The objective in many papers on IE is to show that some innovation leads to better performance than a reasonable baseline. Often this involves the comparison of two or more system variants, at least one of which constitutes the baseline, and one of which embodies the innovation. Typically, the preferred variant achieves the highest scores, if only by small margins, and often this is taken as sufficient evidence of general improvement, even though the test sets in many IE domains are relatively small.

Approximate randomization is a computer-intensive procedure for estimating the statistical significance of a score difference in cases where the predictions of two systems under comparison are aligned at the unit level (Noreen 1989). For example, Chinchor et al. (1993) used this procedure to assess the pairwise separation among participants of MUC3.

Table 5 presents pseudocode for the approximate randomization procedure. The procedure involves a large number (M) of passes through the test set. Each pass involves swapping the baseline and preferred outcomes on approximately half of the test documents, yielding two new “swapped” scores.¹⁸ The fraction of passes for which this procedure widens the gap between systems is an estimate of the p value associated with the observed score difference. If this computed fraction is less than or equal to the desired confidence level (typically 0.05), we are justified in concluding that the observed difference in scores between baseline and preferred is significant.

Table 5 The approximate randomization procedure

In many cases, a relevant baseline is difficult to establish or acquire for the purpose of a paired comparison. Often the most salient comparison is with numbers reported only in the literature. Confidence bounds are critical in such cases to ascertain the level of significance of a result. However, calculating confidence bounds on a score such as the F-measure is cumbersome and possibly dubious, since it is unclear what parametric assumptions to make. Fortunately, we can apply the bootstrap, another computer-intensive procedure, to model the distribution of possible F-measures and assess confidence bounds (Efron and Tibshirani, 1993).

Table 6 sketches this procedure. As in approximate randomization, we iterate a large number (M, typically at least 1000) of times. With each iteration, we calculate the statistic of interest (e.g., the F-measure) on a set of documents from the test set formed by sampling with replacement. The resulting score sample may then be used to assess confidence bounds. In an approach called the percentile bootstrap, these scores are binned by quantile. The upper and lower values of the confidence interval may then be read from this data. For example, the lower bound of the 90% confidence interval lies between the maximum score among the lowest 5% and the next score in an ordering from least to greatest. Obviously, in order for this computation to be valid, M must be sufficiently large. Additional caveats apply, and interested readers are referred to the Efron and Tibshirani introduction (1993).

Table 6 The bootstrap procedure

1.2 Glossary

In the table below, we have listed the names/acronyms of the systems mentioned in the paper together with their full names and bibliographical references.

BIEN	Bayesian Information Extraction Network (Peshkin and Pfeffer 2003)
BWI	Boosted Wrapper Induction (Freitag and Kushmerick 2000)
CProb	Bayesian Prediction Combination (Freitag 1998)
Elie	Adaptive Information Extraction Algorithm (Finn and Kushmerick 2004a, b)
(LP)2	Adaptive Information Extraction Algorithm (Ciravegna 2001a)
ME2	Maximum Entropy Classifier (Chieu and Ng 2002)
PAUM	Perceptron Algorithm with Uneven Margins (Li et al. 2005b)
RAPIER	Robust Automated Production of Information Extraction Rules (Califf 1998)
SNoW	Sparse Network of Winnows (Roth and Yih 2001, 2002)
SRV	Symbolic Relational Learner (Freitag 1998)
SVMUM	Support Vector Machine with Uneven Margins (Li et al. 2005a)
TIES	Trainable Information Extraction System
T-Rex	Trainable Relation Extraction (Iria and Ciravegna 2006)
WHISK	(Soderland 1999)