A methodology for identifying breakthrough topics using structural entropy

Highlights

• Identifying a scientific breakthrough early and helping to establish forward-looking predictions.

• Depicting the non-linear characteristics of complex knowledge networks through structural changes.

• Regarding the knowledge network as a complex system from a holistic perspective.

• Observing the incubation mechanism of emergent scientific breakthroughs from a dynamic evolutionary perspective.

Abstract

This research uses link prediction and structural-entropy methods to predict scientific breakthrough topics. Temporal changes in the structural entropy of a knowledge network can be used to identify potential breakthrough topics. This has been done by tracking and monitoring a network's critical transition points, also known as tipping points. The moment at which a significant change in the structural entropy of a knowledge network occurs may denote the points in time when breakthrough topics emerge. The method was validated by domain experts and was demonstrated to be a feasible tool for identifying scientific breakthroughs early. This method can play a role in identifying scientific breakthroughs and could aid in realizing forward-looking predictions to provide support for policy formulation and direct scientific research.

Introduction

Scientific breakthroughs allow research to be channeled in directions that were previously inaccessible. Thus, such developments are highly innovative and represent the forefront of scientific research. The early identification of breakthrough topics is important for policy formulation and strategic management by governments, businesses, and other organizations. Identifying major breakthroughs early gives policy-makers ample time to react. Compared to incremental innovation, breakthrough innovation is more difficult to identify because it takes time to recognize a development and gather the information needed for analysis. In some cases, luck and intuition play a role in discovering this information, especially when serendipity is involved (van Andel, 1994; Merton et al., 2004; Fukawa, 2006; Seymour, 2009).

Studies have primarily explored the identification of scientific breakthroughs using qualitative methods that depend on expert judgment. However, regarding the background of interdisciplinary integration, relying solely on experts is excessively time-consuming and often leads to contradictory results. An abundance of powerful data-processing resources, analytical tools, and algorithms have emerged that provide effective support for, or alternatives to, expert opinion. Currently, most methods used to identify breakthrough topics rely heavily on specific attributes (e.g., recency, novelty, integrative, knowledge configurability, and word frequency). However, these methods fail to consider processes of gestation, propagation, development, and mutation.)

The early identification of scientific breakthroughs relies on a deep understanding of the laws of technological innovation. Scientific innovation is a nonlinear developmental process, and different knowledge topics interrelate to form complex networks. In these networks, newly emerging topics affect the structure of the original network (Chen, 2012, 2015; Dahlin & Behrens, 2005; Wan, 2017; Xu et al. 2019; Xu et al. 2021; Zhang et al. 2021). Thus, the greater the degree of innovation, the greater is the impact of the topic on the knowledge network structure, thus, scientific breakthroughs always affect the structure of knowledge networks (Luo, 2020). Therefore, evaluating their impact on knowledge-network structures can identify potential breakthrough topics. Structural entropy, considers the number of communities and their sizes, to encapsulate a richer representation of the network's structure into a single value (Almog et al., 2019), conversely, it regards the network holistically and dynamically as a complex system by considering the evolutionary processes (Guan, 2014; Huo, 2019; Xu et al. 2019; Xu et al. 2021), therefore, it would help to identify scientific breakthroughs during their early stages. The increase in the network structure's entropy can be an indicator of the network's evolution, in addition, the growth rate is high during early network evolution but as it grows, the network structure stabilizes, and the growth rate decreases (Luo et al., 2013).

In this study, we attempt to combine link prediction and structural entropy methods to predict scientific breakthrough topics. Compared with extant prediction indicators, the structural entropy index regards the knowledge network as a complex system from a holistic perspective. This method is used to recognize emerging topics that significantly impact the knowledge-network structure and regards them as potential breakthrough topics. Finally, the prediction results of our proposed method are evaluated by domain experts to assess their validity and reliability.

Our objective is to provide a method that allows the detection of scientific breakthroughs in their early stages. The method is based on tracking and monitoring critical points in the evolution of complex knowledge networks to identify potential breakthrough topics. The moment at which a significant change in a network's structural entropy occurs i.e., the tipping point, can be used to denote the point at which breakthrough topics emerge. Therefore, unlike the extant prediction indicators, the structural entropy index holistically regards the knowledge network as a complex system. Our proposed method considers the mechanism of the emergence of scientific breakthroughs from a process perspective, and we believe this is beneficial for the early identification of scientific breakthrough topics. In this respect, our method stands out from most current research as they are inclined toward extant hot-topic monitoring and are not forward-looking.

The remainder of this paper is organized as follows. First, this study details the characteristics of scientific breakthroughs and reviews extant prediction methods. Second, we explain the principles and processes of constructing the prediction model based on structural entropy and link prediction. Then, the field of genetically engineered vaccines (GEV) is used as a testbed, and the prediction results are compared with expert evaluations. Finally, we sum up the advantages and disadvantages and elaborate on future research directions.