Integrating Linguistic and Citation Information with Transformer for Predicting Top-Cited Papers

Abstract

Academic literature contains many different types of data, including language, citations, figures, tables, etc. Released in 2017 for natural language processing, the Transformer model is widely used in fields as diverse as image processing and network science. Transformer models trained on large datasets have been shown to perform well on different tasks when trained on additional small amounts of new data. Most studies of classification and regression in the academic literature have designed individually customized features for specific tasks without fully considering data interactions. Customizing features for each task is costly and complex. This paper addresses this issue by proposing a basic framework that can be consistently applied to various tasks from the diverse information in academic literature. Specifically, we propose an end-to-end fusion method that combines linguistic and citation information of academic literature data utilizing two Transformer models. The experiments were conducted using a dataset on 67 disciplines extracted from the Web of Science, one of the largest databases about academic literature, and classified papers with the top 20% of citations five years after publication. The results show that the proposed method improves the F-measure by 0.028 compared to using only citation or linguistic information on average. Repeated experiments on 67 data sets also showed that the proposed model has the smallest standard deviation of F values. In other words, our proposed method shows the best average performance and is stable with a small variance of the F-value. We also conducted a comparative analysis between the dataset’s characteristics and the proposed method’s performance. The results show that our proposed method correlates poorly with the dataset’s characteristics. In other words, our proposed method is highly versatile. Based on the above, our proposed method is superior regarding F-value, learning stability, and generality.

Appendices

A Queries from Web of Science

We list in Tables 3, 4, 5 and 6 the queries we used to create the dataset from the Web of Science. We use a “Dataset ID” in tables, and this “Dataset ID” is common across all tables. We have multiple queries set up in each dataset. We registered articles matching any one of these queries in the dataset.

Table 3. Queries from Web of Science Dataset 1–20.

Table 4. Queries from Web of Science Dataset 21–40.

Table 5. Queries from Web of Science Dataset 41–60.

Table 6. Queries from Web of Science Dataset 61–70.

B Network and Linguistic Features for Each Dataset

We present the features of each dataset we extracted in Table 7. We targeted a relatively small research area and selected queries As a result, we extracted datasets containing papers from 1,140 to 31,758.

Two types of features are available: one for citation information and the other for language information. The feature on citation information is mainly a network-related indicator. These are the first six (Number of Articles, Number of Nodes, Number of Edges, Network Density, Average Degree, and Gini Coefficient of Degree Distribution). Next, the features related to linguistic information are mainly indicators related to language. These are the last two (Number of Abstracts, Word Perplexity). The Number of Articles represents the number of papers in each dataset. The Number of Nodes represents the number of nodes used in the network, including papers cited in the dataset. The Number of Edges represents the number of edges in the network. The Network Density represents the network density, where N is the number of nodes and E is the number of edges; the network density D calculates as $D = \frac{2E}{N(N-1)}$. The Average Degree represents the average degree, calculated as $\frac{E}{N}$, and the average number of edges per node. The Gini Coefficient of Degree Distribution represents the degree distribution’s Gini coefficient and indicates whether the edges are biased toward a particular node. The Number of Abstracts represents the number of abstracts, which is the number of abstract information included in the data set. The language model may not be fully utilized if this value is low. The Word Perplexity represents the perplexity of words, an indicator of lexical diversity. If the vocabulary in a dataset D is V, and the proportion of occurrences of a word $w_i$ is denoted by $P(w_i)$, then the perplexity PP(D) is calculated by Equation $PP(D) = \Pi _{i=0}^{V} P(w_i)^{-P(w_i)}$.

Table 7. Network and linguistic features for each dataset.

C Result for Each Dataset

We present our results per dataset in Table 8. When we check the results for each dataset, we find that the proposed method only performs the best of 19/67 datasets in the F-value measurement. The “Graph-BERT” method shows the best performance in 34/67 datasets.

Table 8. Classification Results for each dataset.