Abstract
Encompassing an intricately profound propensity for revolutionary, paradigm-shifting ramifications and the potential to wield an irrefutably disruptive sway on forthcoming research endeavors, the notion of the Disruption Index (DI) has surfaced as an object of fervent scientific scrutiny within the realm of scientometrics. Nevertheless, its implementation faces multifaceted constraints. Through a meticulous inquiry, we methodically dissect the limitations of DI, encompassing: (a) susceptibility to variations in reference numbers, (b) vulnerability to intentional author manipulations, (c) heterogeneous manifestations across diverse subject fields, (d) disparities across publication years, (e) misalignment with established scientific impact measures, (f) inadequacy in convergent validity with expert-selected milestones, and (g) a prevalent concentration around zero in its distribution. Unveiling the root causes of these challenges, we propose a viable solution encapsulated in the Rescaled Disruption Index (RDI), achieved through comprehensive rescaling across fields, years, and references. Our empirical investigations unequivocally demonstrate the efficacy of RDI, unveiling the universal nature of disruption distributions in science. This introduces a robust and refined framework for assessing disruptive potential in the scientific landscape while preserving the core principles of the index.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The APS citation dataset is available at https://journals.aps.org/datasets. The Nobel-winning Papers and Laureates data used in this study are open-access at Harvard Dataverse https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/6NJ5RN. For milestone papers identified by the American Physical Society, please visit https://journals.aps.org/125years, https://journals.aps.org/prl/50years/milestones, and https://journals.aps.org/pre/collections/pre-milestones. Other data used in this study can be obtained by making reasonable requests.
References
Arthur, W. B. (2007). The structure of invention. Research Policy, 36(2), 274–287. https://doi.org/10.1016/j.respol.2006.11.005
Arthur, W. B. (2009). The nature of technology: What it is and how it evolves. Simon and Schuster.
Azoulay, P. (2019). Small-team science is beautiful. Nature, 566(7744), 330–332. https://doi.org/10.1038/d41586-019-00350-3
Bak, P., Tang, C., & Wiesenfeld, K. (1987). Self-organized criticality: An explanation of the 1/f noise. Physical Review Letters, 59(4), 381–384. https://doi.org/10.1103/PhysRevLett.59.381
Barabási, A.-L. (2016). Network science. Cambridge University Press.
Barabasi, A. L., & Albert, R. (1999). Emergence of scaling in random networks. Science, 286(5439), 509–512. https://doi.org/10.1126/science.286.5439.509
Betancourt, N., Jochem, T., & Otner, S. M. G. (2023). Standing on the shoulders of giants: How star scientists influence their coauthors. Research Policy, 52(1), 104624. https://doi.org/10.1016/j.respol.2022.104624
Bornmann, L., Devarakonda, S., Tekles, A., & Chacko, G. (2020a). Are disruption index indicators convergently valid? The comparison of several indicator variants with assessments by peers. Quantitative Science Studies, 1(3), 1242–1259. https://doi.org/10.1162/qss_a_00068
Bornmann, L., Devarakonda, S., Tekles, A., & Chacko, G. (2020b). Disruptive papers published in scientometrics: Meaningful results by using an improved variant of the disruption index originally proposed by Wu, Wang, and Evans (2019). Scientometrics, 123(2), 1149–1155. https://doi.org/10.1007/s11192-020-03406-8
Bornmann, L., & Tekles, A. (2019). Disruption index depends on length of citation window. El profesional de la información. https://doi.org/10.3145/epi.2019.mar.07
Bornmann, L., & Tekles, A. (2021). Convergent validity of several indicators measuring disruptiveness with milestone assignments to physics papers by experts. Journal of Informetrics, 15(3), 101159. https://doi.org/10.1016/j.joi.2021.101159
Bower, J. L., & Christensen, C. M. (1995). Disruptive technologies: catching the wave. Harvard Business Review.
Bu, Y., Waltman, L., & Huang, Y. (2021). A multidimensional framework for characterizing the citation impact of scientific publications. Quantitative Science Studies, 2(1), 155–183. https://doi.org/10.1162/qss_a_00109
Chen, J., Shao, D., & Fan, S. (2021). Destabilization and consolidation: Conceptualizing, measuring, and validating the dual characteristics of technology. Research Policy, 50(1), 104115. https://doi.org/10.1016/j.respol.2020.104115
Chu, J. S. G., & Evans, J. A. (2021). Slowed canonical progress in large fields of science. Proceedings of the National Academy of Sciences, 118(41), e2021636118. https://doi.org/10.1073/pnas.2021636118
Cohen, J. (2013). Statistical power analysis for the behavioral sciences. Academic press.
Deng, N., & Zeng, A. (2023). Enhancing the robustness of the disruption metric against noise. Scientometrics, 128(4), 2419–2428. https://doi.org/10.1007/s11192-023-04644-2
Einstein, A. (1916). The foundation of the general theory of relativity. Annalen Phys, 49(7), 769–822.
Fontana, M., Iori, M., Montobbio, F., & Sinatra, R. (2020). New and atypical combinations: An assessment of novelty and interdisciplinarity. Research Policy, 49(7), 104063. https://doi.org/10.1016/j.respol.2020.104063
Fortunato, S., Bergstrom, C. T., Boerner, K., Evans, J. A., Helbing, D., Milojevic, S., Petersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., Vespignani, A., Waltman, L., Wang, D., & Barabasi, A.-L. (2018). Science of science. Science, 359(6379), eaao0185. https://doi.org/10.1126/science.aao0185
Funk, R. J., & Owen-Smith, J. (2017). A dynamic network measure of technological change. Management Science, 63(3), 791–817. https://doi.org/10.1287/mnsc.2015.2366
Gao, J., Zhang, Y.-C., & Zhou, T. (2019). Computational socioeconomics. Physics Reports, 817, 1–104. https://doi.org/10.1016/j.physrep.2019.05.002
Garfield, E., & Sher, I. H. (1963). New factors in the evaluation of scientific literature through citation indexing. American Documentation, 14(3), 195–201. https://doi.org/10.1002/asi.5090140304
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a
Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569–16572. https://doi.org/10.1073/pnas.0507655102
Jo, W. S., Liu, L., & Wang, D. (2022). See further upon the giants: Quantifying intellectual lineage in science. Quantitative Science Studies, 3(2), 319–330. https://doi.org/10.1162/qss_a_00186
Kuhn, T. S. (1962). Historical structure of scientific discovery. Science, 136(3518), 760–764.
Leibel, C., & Bornmann, L. (2023). What do we know about the disruption indicator in scientometrics? An overview of the literature. Preprint https://arxiv.org/abs/2308.02383.
Leydesdorff, L., & Bornmann, L. (2021). Disruption indices and their calculation using web-of-science data: Indicators of historical developments or evolutionary dynamics? Journal of Informetrics, 15(4), 101219. https://doi.org/10.1016/j.joi.2021.101219
Leydesdorff, L., Tekles, A., & Bornmann, L. (2021). A proposal to revise the disruption index. El Profesional De La Información. https://doi.org/10.3145/epi.2021.ene.21
Li, J., Yin, Y., Fortunato, S., & Wang, D. (2019). A dataset of publication records for Nobel laureates. Scientific Data, 6(1), 33. https://doi.org/10.1038/s41597-019-0033-6
Lin, Y., Evans, J. A., & Wu, L. (2022). New directions in science emerge from disconnection and discord. Journal of Informetrics, 16(1), 101234. https://doi.org/10.1016/j.joi.2021.101234
Liu, L., Jones, B. F., Uzzi, B., & Wang, D. (2023). Data, measurement and empirical methods in the science of science. Nature Human Behaviour. https://doi.org/10.1038/s41562-023-01562-4
Mariani, M. S., Medo, M., & Zhang, Y. C. (2016). Identification of milestone papers through time-balanced network centrality. Journal of Informetrics, 10(4), 1207–1223. https://doi.org/10.1016/j.joi.2016.10.005
Mokyr, J. (1990). Punctuated Equilibria and Technological Progress. The American Economic Review, 80(2), 350–354.
Mukherjee, S., Romero, D. M., Jones, B., & Uzzi, B. (2017). The nearly universal link between the age of past knowledge and tomorrow’s breakthroughs in science and technology: The hotspot. Science Advances, 3(4), e1601315. https://doi.org/10.1126/sciadv.1601315
Myers, K. (2020). The elasticity of science. American Economic Journal: Applied Economics, 12(4), 103–134. https://doi.org/10.1257/app.20180518
Park, M., Leahey, E., & Funk, R. J. (2023). Papers and patents are becoming less disruptive over time. Nature, 613(7942), 138–144. https://doi.org/10.1038/s41586-022-05543-x
Price, DJd. S. (1965). Networks of Scientific Papers. Science, 149(3683), 510–515. https://doi.org/10.1126/science.149.3683.510
Radicchi, F., & Castellano, C. (2011). Rescaling citations of publications in physics. Physical Review E, 83(4), 046116. https://doi.org/10.1103/PhysRevE.83.046116
Radicchi, F., Fortunato, S., & Castellano, C. (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences, 105(45), 17268–17272. https://doi.org/10.1073/pnas.0806977105
Ruan, X., Lyu, D., Gong, K., Cheng, Y., & Li, J. (2021). Rethinking the disruption index as a measure of scientific and technological advances. Technological Forecasting and Social Change, 172, 121071. https://doi.org/10.1016/j.techfore.2021.121071
Sheng, L., Lyu, D., Ruan, X., Shen, H., & Cheng, Y. (2023). The association between prior knowledge and the disruption of an article. Scientometrics. https://doi.org/10.1007/s11192-023-04751-0
Shibayama, S., & Wang, J. (2020). Measuring originality in science. Scientometrics, 122(1), 409–427. https://doi.org/10.1007/s11192-019-03263-0
Uzzi, B., Mukherjee, S., Stringer, M., & Jones, B. (2013). Atypical combinations and scientific impact. Science, 342(6157), 468–472. https://doi.org/10.1126/science.1240474
Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365–391. https://doi.org/10.1016/j.joi.2016.02.007
Waltman, L., & van Eck, N. J. (2013). Source normalized indicators of citation impact: An overview of different approaches and an empirical comparison. Scientometrics, 96(3), 699–716. https://doi.org/10.1007/s11192-012-0913-4
Waltman, L., & van Eck, N. J. (2015). Field-normalized citation impact indicators and the choice of an appropriate counting method. Journal of Informetrics, 9(4), 872–894. https://doi.org/10.1016/j.joi.2015.08.001
Wang, D., & Barabási, A.-L. (2021). The science of science. Cambridge University Press.
Wang, D., Song, C., & Barabási, A.-L. (2013). Quantifying long-term scientific impact. Science, 342(6154), 127–132. https://doi.org/10.1126/science.1237825
Wang, J., Veugelers, R., & Stephan, P. (2017). Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy, 46(8), 1416–1436. https://doi.org/10.1016/j.respol.2017.06.006
Wang, S., Ma, Y., Mao, J., Bai, Y., Liang, Z., & Li, G. (2022). Quantifying scientific breakthroughs by a novel disruption indicator based on knowledge entities. Journal of the Association for Information Science and Technology. https://doi.org/10.1002/asi.24719
Wang, W., Liu, Q.-H., Liang, J., Hu, Y., & Zhou, T. (2019). Coevolution spreading in complex networks. Physics Reports, 820, 1–51. https://doi.org/10.1016/j.physrep.2019.07.001
Watson, J. D., & Crick, F. H. C. (1953). Molecular structure of nucleic acids: A structure for deoxyribose nucleic acid. Nature, 171(4356), 737–738. https://doi.org/10.1038/171737a0
Wu, L., Kittur, A., Youn, H., Milojević, S., Leahey, E., Fiore, S. M., & Ahn, Y.-Y. (2022). Metrics and mechanisms: Measuring the unmeasurable in the science of science. Journal of Informetrics, 16(2), 101290. https://doi.org/10.1016/j.joi.2022.101290
Wu, L. F., Wang, D. S., & Evans, J. A. (2019). Large teams develop and small teams disrupt science and technology. Nature, 566(7744), 378. https://doi.org/10.1038/s41586-019-0941-9
Yang, A. J., Deng, S., Wang, H., Zhang, Y., & Yang, W. (2023a). Disruptive coefficient and 2-step disruptive coefficient: Novel measures for identifying vital nodes in complex networks. Journal of Informetrics, 17(3), 101411. https://doi.org/10.1016/j.joi.2023.101411
Yang, A. J., Hu, H., Zhao, Y., Wang, H., & Deng, S. (2023b). From consolidation to disruption: A novel way to measure the impact of scientists and identify laureates. Information Processing & Management, 60(5), 103420. https://doi.org/10.1016/j.ipm.2023.103420
Yang, A. J., Wang, Y., Kong, J., Zhang, Q., Hu, H., Wang, H., & Sanhong, D. (2023). The global disruption index (GDI): an incorporation of citation cascades in the disruptive index. Proceedings of 19th International Society of Scientometrics and Informetrics Conference.
Yang, A. J., Wu, L., Zhang, Q., Wang, H., & Deng, S. (2023d). The k-step h-index in citation networks at the paper, author, and institution levels. Journal of Informetrics, 17(4), 101456. https://doi.org/10.1016/j.joi.2023.101456
Yin, Y., Dong, Y., Wang, K., Wang, D., & Jones, B. F. (2022). Public use and public funding of science. Nature Human Behaviour, 6(10), 1344–1350. https://doi.org/10.1038/s41562-022-01397-5
Yin, Y., & Wang, D. (2017). The time dimension of science: Connecting the past to the future. Journal of Informetrics, 11(2), 608–621. https://doi.org/10.1016/j.joi.2017.04.002
Zeng, A., Shen, Z. S., Zhou, J. L., Wu, J. S., Fan, Y., Wang, Y. G., & Stanley, H. E. (2017). The science of science: From the perspective of complex systems. Physics Reports, 714, 1–73. https://doi.org/10.1016/j.physrep.2017.10.001
Acknowledgements
The authors would like to express gratitude to two anonymous reviewers for their valuable insights. This research was supported by the National Social Science Fund of China (No. 19BTQ062), the Jiangsu Key Laboratory Fund, the International Joint Informatics Laboratory Fund, and the Fundamental Research Funds for the Central Universities.
Author information
Authors and Affiliations
Contributions
AJY: writing-original draft, conceptualization, methodology, software, data visualization, validation, data curation, resources. HG: writing- review and editing, supervision. YW: writing- review and editing, validation. CZ: writing- review and editing, validation. SD: writing-review and editing, supervision, funding acquisition.
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare no competing interest relevant to this article.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Yang, A.J., Gong, H., Wang, Y. et al. Rescaling the disruption index reveals the universality of disruption distributions in science. Scientometrics 129, 561–580 (2024). https://doi.org/10.1007/s11192-023-04889-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-023-04889-x