Tough Tables: Carefully Evaluating Entity Linking for Tabular Data

Abstract

Table annotation is a key task to improve querying the Web and support the Knowledge Graph population from legacy sources (tables). Last year, the SemTab challenge was introduced to unify different efforts to evaluate table annotation algorithms by providing a common interface and several general-purpose datasets as a ground truth. The SemTab dataset is useful to have a general understanding of how these algorithms work, and the organizers of the challenge included some artificial noise to the data to make the annotation trickier. However, it is hard to analyze specific aspects in an automatic way. For example, the ambiguity of names at the entity-level can largely affect the quality of the annotation. In this paper, we propose a novel dataset to complement the datasets proposed by SemTab. The dataset consists of a set of high-quality manually-curated tables with non-obviously linkable cells, i.e., where values are ambiguous names, typos, and misspelled entity names not appearing in the current version of the SemTab dataset. These challenges are particularly relevant for the ingestion of structured legacy sources into existing knowledge graphs. Evaluations run on this dataset show that ambiguity is a key problem for entity linking algorithms and encourage a promising direction for future work in the field.

A 2T Ground Truth Generation Details

1.1 A.1 CEA Table Generation and Preprocessing

2T has been built using real tables. Here we clarify that as a “real table” we intend a table, also artificially built, which resembles a real table. Examples are “list of companies with their market segment”, or “list of Italian merged political parties”, which look like queries that a manager or a journalist could make against a database. The main reasons behind this choice are: (i) it is difficult to get access to real databases; (ii) open data make available a lot of tables, but mostly always tables are in an aggregated form that makes it difficult to annotate them with entities from a general KG like DBpedia. When the data are fine-grained enough, almost all the entities mentioned are not available in the reference KG. For example, in the list of bank failures got from the U.S. Open Data Portal, only 27 over 561 failed banks are represented in DBpedia.

In this section we describe the processes we adopted to collect real tables, or build tables that resembles real ones.

DBpedia Tables. We used the DBpedia SPARQL endpoint as a table generator (SPARQL results are tables). We run queries to generate tables that include:

• entity columns: columns with DBpedia URIs that represent entities.

• “label columns”: columns with possible mentions for the corresponding entities in the entity column. Given an entity column, the corresponding label column has been created by randomly choosing between rdfs:label, foaf:name, or dbo:alias properties.

• literal columns: other columns, with additional information.

Wikipedia Tables. We browsed Wikipedia looking for pages containing tables of interest (e.g., list of presidents, list of companies, list of singers, etc.). We generated different versions of the collected Wikipedia tables, applying different cleaning steps. The following steps have been applied to Wikipedia tables in the TOUGH_MISC category:

• Merged cells have been split in multiple cells with the same value.

• Multi-value cells (slash-separated values, e.g., Pop / Rock, or multi-line values, e.g., Barbados <br> United States, or in-line lists, e.g., <ul>, <li>) have been exploded into several lines. If two or more multi-value cells are on the same line, we exploded all the cells (cartesian product of all the values). If a cell contains the same information in more languages (e.g., anthem song titles), we exploded the cell in two or more columns (the creation of new lines would basically represent duplicates).

Wikipedia tables in the CTRL_WIKI group underwent the next additional cleaning steps:

• “Note”, “Description”, and similar long-text columns have been removed.

• Cells with “None”, “null”, “N/A”, “Unaffiliated”, and similar values have been emptied.

• Columns with only images (e.g., List of US presidents) have been removed.

• All HTML tags have been deleted from cells (e.g., country flag icons);

• Notes, footnotes, and any other additional within-cell information (e.g., birthYear and deathYear for U.S. presidents) have been removed.

Most of all the tables values are already hyperlinked to their Wikipedia page. We used the hyperlinks as the correct annotations (we trust Wikipedia as a correct source of information), following these criteria:

• If a cell content has several links, we took the most relevant annotation, given the column context (e.g., in table https://en.wikipedia.org/wiki/List_of_presidents_of_the_United_States#Presidents the “U.S. senator from Tennessee” cell in the “Prior office” column contains two annotations: https://en.wikipedia.org/wiki/U.S._senator and https://en.wikipedia.org/wiki/Tennessee; in this case we took only the https://en.wikipedia.org/wiki/U.S._senator annotation, as the column is about “Prior offices”, not about places).

• Sometimes it happens that if the same value appears several times in the same column (e.g., music genres), only one instance has the hyperlink to the Wikipedia page. In these cases we copied the same hyperlink to all the instances.

• When the hyperlink is missing (e.g., Hard Rock labels in the table https://en.wikipedia.org/wiki/List_of_best-selling_music_artists#250_million_or_more_records), we manually added the right links by visiting the main entity page (e.g., https://en.wikipedia.org/wiki/Led_Zeppelin) and looking for the missing piece of information (e.g., under the “Genre” section on the Led Zeppelin page we can find Hard Rock linked to https://en.wikipedia.org/wiki/Hard_rock). In case when the information is missing in the main page (e.g., in the same table, Michael Jackson genres include “Dance”, while on his Wiki page the genre is Dance-pop), we manually annotated the value with the most related entity in Wikipedia (in this case, the music genre Dance https://en.wikipedia.org/wiki/Dance_music).

Finally, we converted the Wikipedia links to their DBpedia correspondent links, by replacing https://en.wikipedia.org/wiki/ with http://dbpedia.org/resource in the decoded URL, e.g., https://en.wikipedia.org/wiki/McDonald%27s $\rightarrow$ dbr:McDonald’s, if available, otherwise we manually looked for the right dbpedia link (e.g., https://en.wikipedia.org/wiki/1788-89_United_States_presidential_election $\rightarrow$ dbr:United_States_presidential_election,_1788-89). If this attempt also failed, we left the cell blank (no annotations available in DBpedia).

1.2 A.2 CTA Ground Truth Construction

Automatic CTA Annotations from CEA. The 2T dataset focus is mainly on the entities because, in our opinion, the CEA task is the core task: with good performance in CEA, it is possible to approximate the CTA task easily. We exploited this observation to automatically construct the CTA annotations starting from the CEA ones, which we trust. For each annotated column, we collected all the annotated entities from the CEA dataset and retrieved the most specific type for all the entities from the DBpedia 2016-10 dump.¹⁸ We then annotate the column with the most specific supertype, i.e., the lowest common ancestor of all the types in the DBpedia 2016-10 ontology.¹⁹