WikiDT: Visual-Based Table Recognition and Question Answering Dataset

Abstract

Companies and organizations grapple with the daily burden of document processing. As manual handling is tedious and error-prune, automating this process is a significant goal. In response to this demand, research on table extraction and information extraction from scanned documents in gaining increasing traction. These extractions are fulfilled by machine learning models that require large-scale and realistic datasets for development. However, despite the clear need, acquiring high-quality and comprehensive dataset can be costly. In this work, we introduce the WikiDT, a TableVQA dataset with hierarchical labels for model diagnosis and potentially benefit the research on sub-tasks, e.g. table recognition. This dataset boasts a massive collection of 70,919 images paired with a diverse set of 159,905 tables, providing an extensive corpus for tacking question-answering tasks. The creation of WikiDT is by extending the existing non-synthetic QA datasets, with a fully automated process with verified heuristics and manual quality inspections, and therefore minimizes labeling effort and human errors. A novel focus of WikiDT and its design goal is to answer questions that require locating the target information fragment and in-depth reasoning, given web-style document images. We established the baseline performance on the TableVQA, table extraction, and table retrieval task with recent state-of-the-art models. The results illustrate that WikiDT is yet solved by the existing models that work moderately well on other VQA tasks, and also introduce advanced challenges on table extraction.

Appendices

A Data Acquisition and Processing

1.1 A.1 Overview

WikiDT is created from Wikipedia, a public online encyclopedia created and updated by its users. We are only interested in those Wikipedia web pages with tables, for purpose of table extraction and table-based question answering. Overall, the data acquisition process is taking the Wikipedia URLs from existing datasets, namely WikiTableQuestions [31], TabFact [7], and WikiSQL [46], rendering the web pages to images while annotating the tables, then re-connecting the rendered images and table annotations to the existing question-answer pairs.

The basic annotations in the WikiDT are {image, question, answer} triplets, which are basic input and output to the end-to-end visual-based table question answering models. Aware of the challenges, WikiDT also provides auxiliary annotations include: ground truth table annotations (web tables); Textract table recognition and OCR annotations; QA target table labels based both on the Textract table and web tables; and executable SQL queries on majority of QA samples.

Although the task may seem straightforward, the creation of the dataset is fraught with numerous challenges. Especially, the web contents has changed dramatically since the original WikiSQL dataset was created. Additionally, the data processing in WikiSQL dataset is unknown and irreversible, which poses great hardship in re-connecting the tables to the question-answer pairs. We detail the approaches to overcome those as below.

1.2 A.2 Image Rendering and Pagination

We leverage the Puppeteer² to render the screenshots mainly from the URLs. Generally, the rendered images in the continuous scroll mode have large heights that are extremely hard for the popular visual networks to handle. Thus, the original rendered pages (full-page) are paginated to several sub-pages.

URLs. TabFact provides the URLs to the tables in itself and WikiSQL, and WikiTableQuesions also provide the URLs to its tables. Despite all the URLs are Wikipedia pages, the URLs in the TabFact is Wikipedia domain followed by article title, which changes drastically over the years, while the WikiTableQuestions provided URLs with archived version IDs, which have the exact content when they created the dataset. So for the first step, we search for the Wikipedia editing history for each TabFact pages, and find the closest version to then the WikiSQL dataset is created. Then both of the recovered TabFact pages and WikiTableQuestions pages are rendered.

Renderer Setting. The width of the page is set to 1600 pixels for a better table layout (as compared in Fig. 9). Less than 0.5% of pages have minimum page width requirements that are larger than 1600 pixels, mainly owing to extremely wide tables, and are rendered with their minimum page width.

Pagination. The extra-long full pages cause huge trouble for the down-stream tasks. For example, the typical width-height ratio of input layer to common visual models is 1:1 for VGG [38] and ResNet [13], and other backbone models trained on ImageNet. Therefore the pagination is critical for usage of the dataset. Fixed-height segmentation may divide a single table to multiple subpages. Thus we use a dynamic-height pagination strategy: first, the blank lines are detect via the slide window; concretely, if the color deviations of a W x H window is 0, we consider this window is a blank line. The W is set to 1200 to ignore the navigation bar on the left of the pages, and H is set to 10. Then a subset of blank lines are selected such that the width-to-height ration of every segment, excepting the last one, is close to 1: 1.

1.3 A.3 Table Annotation

The table annotations are derived from HTML table tags, e.g., $\langle$table$\rangle$ translates to a table and $\langle$tr$\rangle$ translates to a table row. The pseudo-code to process the HTML table annotation with Puppeteer is summarized as below. Puppeteer API returns valid bounding box for visible elements and returns NULL for invisible ones. The annotation process keeps only visible tables and table elements. Additionally, the merged table cells are annotated with non-empty row_span and col_span values.

Web Table Filtering. The editors on Wikipedia can use $\langle$table$\rangle$ tag to express a broader range of data (e.g., legend as in Fig. 11). We remove those false tables by criteria that a true table should have at least two rows and two columns. Moreover, the nested tables are possible in web page while hardly seen in other document format. We notice that some web pages in our dataset contains nested tables (e.g., Fig. 7), the outer tables of which are usually used to produce certain layout. Therefore, we detect the nested tables by table bounding boxes and keep only the inner-most tables.

Fig. 7.

Nested tables: the outer tables annotations are removed.

Fig. 8.

Data acquisition for WikiDT table extraction task.

Fig. 9.

Page and table layout in different page width.

Creating the Table Extraction Task. Figure 8 summarizes the data acquisition and processing of WikiDT table extraction dataset. Following the convention in PubTables-1M, we divide the table extraction to table detection which predicts the table bounding boxes from the subpage, and table structure recognition which predicts the inner structure in the table from the table crop. The table extraction task data is labeled using Pascal VOC format.

1.4 A.4 OCR and Textract Table Recognition

Realistically, in Document VQA task, ground truth OCR annotation is unavailable. Instead, datasets usually provide the OCR results from off-the-shelf systems. Following the same principle, the OCR and table recognition from a publicly available system is provided along with the images for Document VQA tasks. We used the AWS Textract service to acquire the OCR and table extraction results on the subpages. In practice, we also found the Textract results quality on the subpages is substantially better than the quality on full pages.³,⁴

1.5 A.5 QA Table Processing

Inspected the WikiSQL and WikiTableQuestion samples with the webpage, there are two observations: 1) the tables to the QA pair could not be the information tables, which are shown on the upper right of the page), and could not be reference tables, which are at the end of the page; 2) the tables to the QA pairs can not have only one content row. Therefore, we remove those tables in the QA task.

Furthermore, the content of tables are normalized in the following ways. First, we remove the JavaScript code that are extracted as texts by Puppeteer, which are detected by regular expression matching to keywords (e.g., mw-parser-output, navbar). Then, if any cell text is longer than 200 characters, it is replaced with a special token $\langle {TLDR}\rangle$. Lastly, the string normalization method adopted in WikiTableQuestions is applied to each table cell text.

Beyond table filtering and table content normalization, the table headers, if contains more than one rows, are flattened. If the table headers are row-headers and multi-level, the headers are transformed using level1.level2... format, except for the level where the text for all the cells in that level is the same, i.e., a uniform level value. If the table has only column header, the table are transposed. If the table is 2D, only the row header is considered as header.

1.6 A.6 Table Retrieval Labels

Table retrieval is an essential step if there are many tables on the page. The table retrieval labels are generated automatically, primarily by retrieving the table with the highest recall using the reference tables and answers from the original dataset (i.e., WikiTableQuestions and WikiSQL) as queries. The process is illustrated in Fig. 10. Note that the retrieval label generation process does not provide any cheating method for the table retrieval task: the label generation is based on a reference table, which is not included in the WikiDT dataset and should be assumed unknown to the model, and the answer, which is also unknown to the retrieval model. The table retrieval model of the user of WikiDT should use only the question as a query.

Fig. 10.

Data acquisition and processing for WikiDT table retrieval sub-task.

Algorithm 1 shows the table retrieval label generation process. The reference tables and answers from WikiSQL and WikiTableQuestions serve as queries and the candidate tables are regarded as keys. We record the retrieved table by best recall to the reference table and reference keys. In most cases (74%), the two results agree. For the QA experiments, we use the best recall to the reference table results, since the reference answers may not appear in the table owing to the aggregation functions.

Fig. 11.

False table (upper left one) and true table (lower).

1.7 A.7 Answer Generation

Inspecting the table retrieval results from web table annotations, we found that the contents of tables in the WikiSQL dataset are crucially different from the contents on the images, presumably due to the web content updates and post-processing during the creation of the WikiSQL dataset. Therefore, we translated the SQL query from the WikiSQL dataset and executed it on the retrieved web table in WikiDT to generate the answers to the questions, shown in Fig. 12. On the other hand, the tables in WikiTableQuestions do not have this issue, thus the answers are still valid given the images and the questions.

Fig. 12.

QA pair generation from WikiSQL dataset from acquired images and tables.

B Compare to Existing VQA Datasets

Table 2 compare the input modality and task formulation of the notable VQA datasets, and a concrete examples of the VQA samples are shown in Fig. 13. It can be clearly seen that WikiDT provides a unique type of QA tasks, and are highly challenging in the plenitude of the context information.

Fig. 13.

Illustration of VQA datasets

C Dataset Description and Analysis

1.1 C.1 Tasks and Datasets

WikiDT-Detection Dataset. With the results of pagination, we take the subpage images and the table bounding boxes, ignoring the table structures and content, as a table detection dataset. Images with no tables are discarded.

WikiDT-Structure Dataset. Independent with the pagination, each table region is cropped from the full-page images to form the image inputs to the table structure recognition task. The coordinates of the bounding boxes are translated from full-page coordinates to the cropped image coordinates. The following types of structural objects are annotated directly from raw web table annotation: table body rows, header rows, table cells, and merged cells. Since the HTML tags do not have table column tags while columns are essential labels for many existing table extraction models (e.g., TableNet [28]), we compute the column bounding boxes and include them in the annotation as well.

WikiDT-TableVQA Dataset. The question answering task is to predict the answer to a question given the full page image in an end-to-end manner. Meanwhile, the intermediate labels along the reasoning chains are available for breaking down and diagnosing the models. The intermediate labels include table annotation, table retrieval labels, and SQL queries in most of the samples. The OCR results from Textract are highly accurate and could be leveraged directly to fit into existing document VQA or scene-text VQA frameworks.

1.2 C.2 Data Analysis

Images and Subpages. Figure 14 shows the image height distribution before (full page) and after (sub-page) pagination step. It shows the pagination greatly reduced average image heights and extremely long pages. The outliers that still have large image heights may contain long tables that the pagination module is unable to split. Figure 15 shows that the number of tables per image reduces after pagination.

Fig. 14.

Image Heights Distributions (x-axis in $10^3$).

Page Layout. Comparing to existing table recognition dataset created from PDF, the detection task of WikiDT is challenging with diverse table shapes and positions, illustrated in Fig. 17. PubTables-1M images are single or double columned PDFs. The table position and width, especially in the double-columned documents are highly similar. On the contrary, table location and shape are much more arbitrary in WikiDT. Figure 16 plots the distribution of table bounding boxes of both dataset. It’s obviously seen that the PubTables-1M has few spikes indicating the few locations that are highly likely to be the table border, while the distribution in WikiDT are almost uniform which indicates a diversity in the table location. Moreover, WikiDT contains list-like text blocks that could be easily mis-detected as tables.

Fig. 15.

Number of tables on a single image.

Fig. 16.

Table location distribution.

Table 10. Textract table recognition accuracy.

Fig. 17.

Table detection samples from PubTables-1M (upper) and WikiDT (lower).

Table Styles. The table structure recognition task is featured with diversified table styles, shown in Fig. 18.

Qualitative Annotation Results from Textract. AWS Textract gives overall high quality table recognition results, but there are still wrong recognition cases shown in Fig. 19. In the experiment section, we compare the two approaches that utilize the Textract Table recognition results with TAPAS model, and another one uses the OCR boxes. Though the TAPAS model has overall better performance, its function relies on the correct recognition results and could not learn to rectify the table recognition error, like the one in Fig. 19 (4), when the two columns are recognized as one.

Fig. 18.

Various types of table headers.

Top Answer Items. Table 11 shows the most frequent numeric answers and the text answers. Usually, the numeric answers are produced from some aggregations. For instance, count the number of rows that satisfy certain criteria, or comparing the difference between two values (see Fig. 22 for example). To this end, understanding the implied reasoning from the questions is critical to solve the QA task.

Table 11. Top answer items from WikiDT-TableQA/VQA task.

D Additional Table Recognition Results

Figure 19, 20, 21 illustrate the qualitative table detection and table recognition results. Generally, the DETR model and Textract can generate reliable results, despite that viewing from Table 8 the table extraction performance still have potential to improve.

Fig. 19.

Textract recognition results. Red boxes show OCR tokens, olive boxes show table structure. (Color figure online)

Fig. 20.

DETR prediction compared with ground truth.

Fig. 21.

Table extraction result comparison.

Fig. 22.

TAPAS prediction examples (left: correct, right: wrong). Light blocks show the column selection result and the darker blocks show the cell selection.

E TableVQA Additional Results

Figure 22 shows when TAPAS makes correct and wrong predictions. Notice that in the wrong prediction case, the question requires a multi-step aggregation. The QA system needs to count the number of ships wrecked in Lake Huron and in Lake Erie separately, then compare the difference. However, by design, TAPAS could solve questions with no more than one aggregation.