Abstract
Governments typically formulate sets of policies to guide the direction of scientific research. And the possible effects of these policies on scientific research have been explored. However, there is limited literature reflecting how policy information is incorporated and concerned by scientific research. In this research, we explore policy usages in scientific research by analyzing the occasions when these policies are mentioned. Taking China’s S&T evaluation policies as a specific case, we analyzed mentions of these policies in terms of velocity, strength, usage priority, usage variations, and usage intentions. From these perspectives, we found that the strength and velocity of mentions have consistently increased over the years. Additionally, researchers have tended to focus more on policies issued by senior agencies or by multiple agencies. Further, there are great variations in policy sensitivity across institutions and disciplines, with universities actively reacting to policies and social science research showing remarkable policy take-up. Last, these policies are often mentioned in the analysis section of papers alongside the major functionality of policy content reference. However, the locations and specific usages of the policies do not follow a fixed model. Overall, the quantitative side of our analysis reveals the diverse policy usages in scientific research activity. These new insights sharpen our perceptions of how policies pertain to and are used by scientific research in real-world settings.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Basit, T. N. (2003). Manual or electronic? The role of coding in qualitative data analysis. Educational Research, 45(2), 143–154. https://doi.org/10.1080/0013188032000133548
Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8(4), 895–903. https://doi.org/10.1016/J.JOI.2014.09.005
Bornmann, L., & Haunschild, R. (2017). Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact. Scientometrics, 110(2), 937–943. https://doi.org/10.1007/S11192-016-2200-2
Cherney, A., Head, B., Boreham, P., Povey, J., & Ferguson, M. (2013). Research Utilization in the Social Sciences: A comparison of five academic disciplines in Australia. Science Communication, 35(6), 780–809. https://doi.org/10.1177/1075547013491398
Ciarli, T., & Ràfols, I. (2019). The relation between research priorities and societal demands: The case of rice. Research Policy, 48(4), 949–967. https://doi.org/10.1016/J.RESPOL.2018.10.027
Debackere, K., & Glänzel, W. (2004). Using a bibliometric approach to support research policy making: The case of the Flemish BOF-key. Scientometrics, 59(2), 253–276. https://doi.org/10.1023/B:SCIE.0000018532.70146.02
Fang, Z., & Costas, R. (2020). Studying the accumulation velocity of altmetric data tracked by Altmetric.com. Scientometrics, 123(2), 1077–1101. https://doi.org/10.1007/S11192-020-03405-9
Fang, Z., Dudek, J., Noyons, E., & Costas, R. (2020). Science cited in policy documents: Evidence from the Overton database. In Altmetrics conference. http://altmetrics.org/wpcontent/uploads/2020/11/02_submission_Fang_Dudek_Noyons_Costasaltmetrics20.pdf.
Franzoni, C., Scellato, G., & Stephan, P. (2011). Changing incentives to publish. Science, 333(6043), 702–703. https://doi.org/10.1126/SCIENCE.1197286
Furman, J. L., Murray, F., & Stern, S. (2012). Growing stem cells: The impact of federal funding policy on the U.S. scientific frontier. Journal of Policy Analysis and Management, 31(3), 661–705. https://doi.org/10.1002/PAM.21644
Garfield, E., & Welljams-Dorof, A. (1992). Citation data: Their use as quantitative indicators for science and technology evaluation and policy-making. Science and Public Policy, 19(5), 321–327. https://doi.org/10.1093/SPP/19.5.321
Gu, J. (2016). Spatial diffusion of social policy in China: Spatial convergence and neighborhood interaction of vocational education. Applied Spatial Analysis and Policy, 9(4), 503–527. https://doi.org/10.1007/S12061-015-9161-3
Haunschild, R., & Bornmann, L. (2017). How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data. Scientometrics, 110(3), 1209–1216. https://doi.org/10.1007/S11192-016-2237-2
Haustein, S., Costas, R., & Larivière, V. (2015). Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE. https://doi.org/10.1371/JOURNAL.PONE.0120495
Huang, C., & Sharif, N. (2016). Global technology leadership: The case of China. Science and Public Policy, 43(1), 62–73. https://doi.org/10.1093/SCIPOL/SCV019
Huang, C., Su, J., Xie, X., Ye, X., Li, Z., Porter, A., & Li, J. (2015). A bibliometric study of China’s science and technology policies: 1949–2010. Scientometrics, 102(2), 1521–1539. https://doi.org/10.1007/S11192-014-1406-4
Huang, Y., Li, R., Zhang, L., & Sivertsen, G. (2021). A comprehensive analysis of the journal evaluation system in China. Quantitative Science Studies, 2(1), 300–326. https://doi.org/10.1162/QSS_A_00103
Jong, S. P. L. D., Smit, J., Drooge, L., & v. (2016). Scientists’ response to societal impact policies: A policy paradox. Science and Public Policy, 43(1), 102–114. https://doi.org/10.1093/SCIPOL/SCV023
Karmakar, M., Banshal, S. K., & Singh, V. K. (2021). A large-scale comparison of coverage and mentions captured by the two altmetric aggregators: Altmetric.com and PlumX. Scientometrics. https://doi.org/10.1007/S11192-021-03941-Y
Kerkhoff, L. E. V., & Lebel, L. (2015). Coproductive capacities: Rethinking science-governance relations in a diverse world. Ecology and Society. https://doi.org/10.5751/ES-07188-200114
Kishi, N. (2020). How does policy focus influence scientific research. Science and Public Policy, 47(1), 114–124. https://doi.org/10.1093/SCIPOL/SCZ051
Korytkowski, P., & Kulczycki, E. (2019). Examining how country-level science policy shapes publication patterns: The case of Poland. Scientometrics, 119(3), 1519–1543. https://doi.org/10.1007/S11192-019-03092-1
Korytkowski, P., & Kulczycki, E. (2021). The gap between Plan S requirements and grantees’ publication practices. Journal of Informetrics. https://doi.org/10.1016/J.JOI.2021.101156
Laudel, G. (2006). The art of getting funded: How scientists adapt to their funding conditions. Science and Public Policy, 33(7), 489–504. https://doi.org/10.3152/147154306781778777
Martin, B. (2003). The changing social contract for science and the evolution of the university. Science and innovation: Rethinking the rationales for funding and governance (pp. 7–29). Edward Elgar, Cheltenham.
Matthies, H., & Torka, M. (2019). Academic habitus and institutional change: Comparing two generations of German scholars. Minerva, 57(3), 345–371. https://doi.org/10.1007/s11024-019-09370-9
Meulen, BVd. (1998). Science policies as principal–agent games: Institutionalization and path dependency in the relation between government and science. Research Policy, 27(4), 397–414. https://doi.org/10.1016/S0048-7333(98)00049-3
Schneider, J. W., Aagaard, K., & Bloch, C. W. (2016). What happens when national research funding is linked to differentiated publication counts? A comparison of the Australian and Norwegian publication-based funding models. Research Evaluation, 25(3), 244–256. https://doi.org/10.1093/RESEVAL/RVV036
Shu, F., Julien, C.-A., Zhang, L., Qiu, J., Zhang, J., & Larivière, V. (2019). Comparing journal and paper level classifications of science. Journal of Informetrics, 13(1), 202–225. https://doi.org/10.1016/J.JOI.2018.12.005
Tosun, J., & Lang, A. (2017). Policy integration: Mapping the different concepts. Policy Studies, 38(6), 553–570. https://doi.org/10.1080/01442872.2017.1339239
Vilkins, S., & Grant, W. J. (2017). Types of evidence cited in Australian Government publications. Scientometrics, 113(3), 1681–1695. https://doi.org/10.1007/S11192-017-2544-2
Yin, Y., Gao, J., Jones, B. F., & Wang, D. (2021). Coevolution of policy and science during the pandemic. Science, 371(6525), 128–130. https://doi.org/10.1126/SCIENCE.ABE3084
Acknowledgements
This work was supported by the National Natural Science Foundation of China (Grant Nos. 72004169, 71974150, 71974094), the National Social Science Foundation of China (Grant No. 18VSJ087), and the National Laboratory Center for Library and Information Science in Wuhan University.
Author information
Authors and Affiliations
Corresponding authors
Appendix
Rights and permissions
About this article
Cite this article
Li, Y., Mao, J., Zhang, L. et al. How scientific research incorporates policy: an examination using the case of China’s science and technology evaluation system. Scientometrics 127, 5283–5306 (2022). https://doi.org/10.1007/s11192-021-04215-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-021-04215-3