Skip to main content
SpringerLink
Log in

The funding factor: a cross-disciplinary examination of the association between research funding and citation impact

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

This paper intends to illuminate the relationship between science funding and citation impact in seven STEMM disciplines (science, technology, engineering, mathematics, and medicine). Using a regression model with Heckman bias correction, we find that funding has a positive, significant association with a paper’s citations in STEMM fields. Further analyses show that this association is magnified by the factors of multiple authorship and multiple institutions. For funded papers in STEM, multi-author and multi-institution papers tend to receive even more citations than single-authored and single-institution papers; however, funded papers in Medicine received less gain in citation impact when either factor is considered. Based on the finding that funding support has a stronger association with citation impact when it is treated as a binary variable than as a count variable, this paper recommends the allocation of funding to researchers without active funding support, instead of giving awards to those with multiple funding supports at hand.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1.  In our original data collection, we also included several social science and humanity domains; however, during analyses, we realized that the percentage of papers with funding acknowledgement in these domains was rather inconsistent. We believe this is an artifact caused by the inconsistent coverage of the database, as pointed out by (Álvarez-Bornstein et al. 2017; Tang et al. 2017) and decided to only focus on STEMM domains.

  2.  Hirst’s selection method did not converge for Environmental Studies because of the interdisciplinary nature of this domain. Instead, we manually picked five journals based on the citation relations of Nature Climate Change and Global Environmental Change.

References

  • Álvarez-Bornstein, B., Morillo, F., & Bordons, M. (2017). Funding acknowledgments in the Web of Science: Completeness and accuracy of collected data. Scientometrics, 112(3), 1793–1812.

    Article  Google Scholar 

  • Azoulay, P., Stuart, T., & Wang, Y. (2013). Matthew: Effect or fable? Management Science, 60(1), 92–109.

    Article  Google Scholar 

  • Borsuk, R., Budden, A., Leimu, R., Aarssen, L., & Lortie, C. (2009). The influence of author gender, national language and number of authors on citation rate in ecology. Open Ecology Journal, 2, 25–28.

    Article  Google Scholar 

  • Boyack, K. W., & Börner, K. (2003). Indicator-assisted evaluation and funding of research: Visualizing the influence of grants on the number and citation counts of research papers. Journal of the American Society for Information Science and Technology, 54(5), 447–461.

    Article  Google Scholar 

  • Boyack, K. W., & Klavans, R. (2015). Is the most innovative research being funded. Paper presented at the 20th international conference on science and technology indicators.

  • Bushway, S., Johnson, B. D., & Slocum, L. A. (2007). Is the magic still there? The use of the Heckman two-step correction for selection bias in criminology. Journal of Quantitative Criminology, 23(2), 151–178.

    Article  Google Scholar 

  • Campbell, D., Picard-Aitken, M., Côté, G., Caruso, J., Valentim, R., Edmonds, S., et al. (2010). Bibliometrics as a performance measurement tool for research evaluation: The case of research funded by the National Cancer Institute of Canada. American Journal of Evaluation, 31(1), 66–83.

    Article  Google Scholar 

  • Certo, S. T., Busenbark, J. R., Woo, H. S., & Semadeni, M. (2016). Sample selection bias and Heckman models in strategic management research. Strategic Management Journal, 37(13), 2639–2657.

    Article  Google Scholar 

  • Costas, R., & Leeuwen, T. N. (2012). Approaching the “reward triangle”: General analysis of the presence of funding acknowledgments and “peer interactive communication” in scientific publications. Journal of the American Society for Information Science and Technology, 63(8), 1647–1661.

    Article  Google Scholar 

  • Cragin, M. H., Nichols, L., Simon, M., & Watts, S. M. (2012). Measuring science: Emerging tools for analysis of federal R&D investments. In Proceedings of the ASIST annual meeting, Vol. 49(1).

  • Cronin, B. (1984). The citation process. The role and significance of citations in scientific communication. London: Taylor Graham.

    Google Scholar 

  • Cronin, B., & Shaw, D. (1999). Citation, funding acknowledgement and author nationality relationships in four information science journals. Journal of Documentation, 55(4), 402–408.

    Article  Google Scholar 

  • Dennis, B., & Eilperin, J. (2017). Trump administration tells EPA to freeze all grants, contracts. Washington Post. Retrieved from https://www.washingtonpost.com/news/energy-environment/wp/2017/01/23/trump-administration-tells-epa-to-freeze-all-grants-contracts/?utm_term=.1c3f8ef75a0d.

  • Figg, W. D., Dunn, L., Liewehr, D. J., Steinberg, S. M., Thurman, P. W., Barrett, J. C., et al. (2006). Scientific collaboration results in higher citation rates of published articles. Pharmacotherapy: The Journal of Human Pharmacology and Drug Therapy, 26(6), 759–767.

    Article  Google Scholar 

  • Fortin, J.-M., & Currie, D. J. (2013). Big science vs. little science: How scientific impact scales with funding. PLoS ONE, 8(6), e65263.

    Article  Google Scholar 

  • Franceschet, M., & Costantini, A. (2010). The effect of scholar collaboration on impact and quality of academic papers. Journal of Informetrics, 4(4), 540–553.

    Article  Google Scholar 

  • Gök, A., Rigby, J., & Shapira, P. (2016). The impact of research funding on scientific outputs: Evidence from six smaller European countries. Journal of the Association for Information Science and Technology, 67(3), 715–730.

    Article  Google Scholar 

  • Grassano, N., Rotolo, D., Hutton, J., Lang, F., & Hopkins, M. M. (2017). Funding data from publication acknowledgments: Coverage, uses, and limitations. Journal of the Association for Information Science and Technology, 68(4), 999–1017.

    Article  Google Scholar 

  • Heckman, J. J. (1976). The common structure of statistical models of truncation, sample selection and limited dependent variables and a simple estimator for such models. In Annals of economic and social measurement (Vol. 5, pp. 475–492): NBER.

  • Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica: Journal of the Econometric Society, 47, 153–161.

    Article  MathSciNet  MATH  Google Scholar 

  • Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., & Rafols, I. (2015). The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431.

    Article  Google Scholar 

  • Hirst, G. (1978). Discipline impact factors: A method for determining core journal lists. Journal of the American Society for Information Science, 29(4), 171–172.

    Article  Google Scholar 

  • Holton, G. (1978). Can science be measured. Toward a metric of science: The advent of science indicators. New York: Social Science Research Council.

    Google Scholar 

  • Jacob, B. A., & Lefgren, L. (2011). The impact of research grant funding on scientific productivity. Journal of public economics, 95(9), 1168–1177.

    Article  Google Scholar 

  • Lane, J. (2009). Assessing the impact of science funding. Science, 324(5932), 1273–1275.

    Google Scholar 

  • Lane, J., & Bertuzzi, S. (2010). The STAR METRICS project: Current and future uses for S&E workforce data. Paper presented at the Science of Science Measurement Workshop, Held in Washington DC.

  • Lane, J., & Bertuzzi, S. (2011). Measuring the results of science investments. Science, 331(6018), 678–680.

    Article  Google Scholar 

  • Lane, J., Owen-Smith, J., Rosen, R. F., & Weinberg, B. A. (2015). New linked data on research investments: Scientific workforce, productivity, and public value. Research Policy, 44(9), 1659–1671.

    Article  Google Scholar 

  • Largent, M. A., & Lane, J. I. (2012). STAR METRICS and the science of science policy. Review of Policy Research, 29(3), 431–438.

    Article  Google Scholar 

  • Larivière, V., & Gingras, Y. (2010). The impact factor’s Matthew effect: A natural experiment in bibliometrics. Journal of the American Society for Information Science and Technology, 61(2), 424–427.

    Google Scholar 

  • Leydesdorff, L., & Wagner, C. (2009). Macro-level indicators of the relations between research funding and research output. Journal of Informetrics, 3(4), 353–362.

    Article  Google Scholar 

  • Ma, A., Mondragón, R. J., & Latora, V. (2015). Anatomy of funded research in science. Proceedings of the National Academy of Sciences, 112(48), 14760–14765.

    Article  Google Scholar 

  • Merton, R. K. (1968). The Matthew effect in science. Science, 159(3810), 56–63.

    Article  Google Scholar 

  • Mervis, J. (2015). After 2-year battle, House passes COMPETES Act on mostly party-line vote. Science. Retrieved November 14, 2017, from http://www.sciencemag.org/news/2015/05/after-2-year-battle-house-passes-competes-act-mostly-party-line-vote.

  • National Science Board. (n.d.). Science and engineering indicators 2016. Retrieved November 14, 2017, from https://www.nsf.gov/statistics/2016/nsb20161/.

  • Park, H., Lee, J. J., & Kim, B.-C. (2015). Project selection in NIH: A natural experiment from ARRA. Research Policy, 44(6), 1145–1159.

    Article  Google Scholar 

  • Paul-Hus, A., Desrochers, N., & Costas, R. (2016). Characterization, description, and considerations for the use of funding acknowledgement data in Web of Science. Scientometrics, 108(1), 167–182.

    Article  Google Scholar 

  • Rigby, J. (2013). Looking for the impact of peer review: Does count of funding acknowledgements really predict research impact? Scientometrics, 94(1), 57–73.

    Article  Google Scholar 

  • Sarli, C. C., & Carpenter, C. R. (2014). Measuring academic productivity and changing definitions of scientific impact. Missouri Medicine, 111(5), 399.

    Google Scholar 

  • Tang, L., Hu, G., & Liu, W. (2017). Funding acknowledgment analysis: Queries and Caveats. Journal of the Association for Information Science and Technology, 68(3), 790–794.

    Article  Google Scholar 

  • Trochim, W. M., Marcus, S. E., Mâsse, L. C., Moser, R. P., & Weld, P. C. (2008). The evaluation of large research initiatives a participatory integrative mixed-methods approach. American Journal of Evaluation, 29(1), 8–28.

    Article  Google Scholar 

  • Van Noorden, R. (2015). Seven thousand stories capture impact of science. Nature, 518(7538), 150.

    Article  Google Scholar 

  • Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. (2011). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics, 5(1), 37–47.

    Article  Google Scholar 

  • Wang, X., Liu, D., Ding, K., & Wang, X. (2011). Science funding and research output: A study on 10 countries. Scientometrics, 91(2), 591–599.

    Article  Google Scholar 

  • Wang, J., & Shapira, P. (2011). Funding acknowledgement analysis: An enhanced tool to investigate research sponsorship impacts: The case of nanotechnology. Scientometrics, 87(3), 563–586.

    Article  Google Scholar 

  • Weinberg, B. A., Owen-Smith, J., Rosen, R. F., Schwarz, L., Allen, B. M., Weiss, R. E., et al. (2014). Science funding and short-term economic activity. Science, 344(6179), 41–43.

    Article  Google Scholar 

  • Wooldridge, J. M. (2015). Introductory econometrics: A modern approach. New York: Nelson Education.

    Google Scholar 

  • Yan, E., & Zhu, Y. (2015). Identifying entities from scientific publications: A comparison of vocabulary-and model-based methods. Journal of Informetrics, 9(3), 455–465.

    Article  MathSciNet  Google Scholar 

  • Yong, E. (2017). Thanks to trump, scientists are planning to run for office. The Atlantic. Retrieved from https://www.theatlantic.com/science/archive/2017/01/thanks-to-trump-scientists-are-planning-to-run-for-office/514229/.

  • Zhao, D. (2010). Characteristics and impact of grant-funded research: A case study of the library and information science field. Scientometrics, 84(2), 293–306.

    Article  Google Scholar 

Download references

Acknowledgements

This project was made possible in part by the Institute of Museum and Library Services (Grant Award Number: RE-07-15-0060-15), for the project titled “Building an entity-based research framework to enhance digital services on knowledge discovery and delivery”. This work was also partly supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2015S1A3A2046711).

Author information

Authors and Affiliations

  1. College of Computing and Informatics, Drexel University, 3141 Chestnut Street, Philadelphia, PA, 19104, USA

    Erjia Yan

  2. LeBow College of Business, Drexel University, 3141 Chestnut Street, Philadelphia, PA, 19104, USA

    Chaojiang Wu

  3. Department of Library and Information Science, Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul, 03722, Korea

    Min Song

Authors
  1. Erjia Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chaojiang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Min Song
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Erjia Yan.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 21 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, E., Wu, C. & Song, M. The funding factor: a cross-disciplinary examination of the association between research funding and citation impact. Scientometrics 115, 369–384 (2018). https://doi.org/10.1007/s11192-017-2583-8

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-017-2583-8

Keywords

Search

Navigation