Skip to main content
SpringerLink
Log in

Validating online reference managers for scholarly impact measurement

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

This paper investigates whether CiteULike and Mendeley are useful for measuring scholarly influence, using a sample of 1,613 papers published in Nature and Science in 2007. Traditional citation counts from the Web of Science (WoS) were used as benchmarks to compare with the number of users who bookmarked the articles in one of the two free online reference manager sites. Statistically significant correlations were found between the user counts and the corresponding WoS citation counts, suggesting that this type of influence is related in some way to traditional citation-based scholarly impact but the number of users of these systems seems to be still too small for them to challenge traditional citation indexes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aguillo, I. (2011). Is Google Scholar useful for bibliometrics? A webometrics analysis. Proceedings of the ISSI 2011 Conference. Presented at the 13th International Conference of the International Society for Scientometrics & Informetrics (pp. 13–18), Durban, 4–7 July 2011.

  • Belew, R. K. (2005). Scientific impact quantity and quality: Analysis of two sources of bibliographic data. Arxiv preprint cs/0504036.

  • Brody, T., Harnad, S., & Carr, L. (2006). Earlier web usage statistics as predictors of later citation impact. Journal of the American Society for Information Science and Technology, 57(8), 1060–1072.

    Article  Google Scholar 

  • Burgelman, J., Osimo, D., & Bogdanowicz, M. (2010). Science 2.0 (change will happen…). First Monday, 15(7). Retrieved from http://www.uic.edu/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/2961/2573.

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Mahwah: Lawrence Erlbaum.

  • Cronin, B. (2001). Bibliometrics and beyond: Some thoughts on web-based citation analysis. Journal of Information Science, 27(1), 1–7.

    Article  Google Scholar 

  • Cronin, B., Snyder, H. W., Rosenbaum, H., Martinson, A., & Callahan, E. (1998). Invoked on the web. Journal of the American Society for Information Science, 49(14), 1319–1328. doi:10.1002/(SICI)10974571(1998)49:14<1319:AID-ASI9>3.0.CO;2-W.

    Article  Google Scholar 

  • Ginsparg, P. (2007). Next-generation implications of Open Access. CTWatch Quarterly, 2(3). Retrieved from http://www.ctwatch.org/quarterly/print.php?p=80.

  • Groth, P., & Gurney, T. (2010). Studying scientific discourse on the Web using bibliometrics: A chemistry blogging case study—web science repository. Proceedings of the WebSci’10. Raleigh, NC. Retrieved April 26–27, 2010, from http://journal.webscience.org/308/.

  • Henning, V. (2010). The top 10 journal articles published in 2009 by readership on Mendeley | Mendeley Blog. Retrieved August 8, 2010, from http://www.mendeley.com/blog/academic-features/the-top-10-journal-articles-published-in-2009-by-readership-on-mendeley/.

  • Henning, V., & Reichelt, J. (2008). Mendeley—A Last.fm for research? In IEEE Fourth International Conference on eScience, 2008 (pp. 327–328).

  • Jiang, J., He, D., & Ni, C. (2011). Social reference: Aggregating online usage of scientific literature in CiteULike for clustering academic resources. Proceeding of the 11th annual international ACM/IEEE joint conference on Digital libraries (pp. 401–402), Ottawa, Ontario, 13–17 June 2011.

  • Kousha, K., & Thelwall, M. (2007). Google Scholar citations and Google Web/URL citations: A multi-discipline exploratory analysis. Journal of the American Society for Information Science and Technology, 58(7), 1055–1065.

    Article  Google Scholar 

  • Kousha, K., & Thelwall, M. (2008). Assessing the impact of disciplinary research on teaching: An automatic analysis of online syllabuses. Journal of the American Society for Information Science and Technology, 59(13), 2060–2069.

    Article  Google Scholar 

  • Kousha, K., & Thelwall, M. (2009). Google book search: Citation analysis for social science and the humanities. Journal of the American Society for Information Science and Technology, 60(8), 1537–1549.

    Article  Google Scholar 

  • Kousha, K., Thelwall, M., & Rezaie, S. (2010). Using the Web for research evaluation: The Integrated Online Impact indicator. Journal of Informetrics, 4(1), 124–135. doi:10.1016/j.joi.2009.10.003.

    Article  Google Scholar 

  • Li, X., Thelwall, M., & Giustini, D. (2011). Validating online reference managers for scholarly impact measurement. In E. Noyons, P. Ngulube & J. Leta (Eds.), Proceedings of ISSI 2011—The 13th International Conference on Scientometrics and Informetrics (pp. 454–462), Durban, 4–7 July 2011.

  • Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105–2125.

    Article  Google Scholar 

  • Mendeley. (2010). Academic reference management software for researchers | Mendeley. Retrieved August 8, 2010, from http://www.mendeley.com/.

  • Moed, H. F. (2005). Citation analysis in research evaluation. New York: Springer.

    Google Scholar 

  • Neylon, C., & Wu, S. (2009). Article-level metrics and the evolution of scientific impact. PLoS Biol, 7(11), e1000242. doi:10.1371/journal.pbio.1000242.

    Article  Google Scholar 

  • Norris, M., & Oppenheim, C. (2010). Peer review and the h-index: Two studies. Journal of Informetrics, 4(3), 221–232. doi:10.1016/j.joi.2009.11.001.

    Article  Google Scholar 

  • PLoS ONE. (2009a). Article-level metrics. Retrieved July 20, 2010, from http://article-level-metrics.plos.org/.

  • PLoS ONE. (2009b). New addition to article-level metrics—blog posts from ResearchBlogging.org | Public Library of Science. Retrieved July 20, 2010, from http://www.plos.org/cms/node/500.

  • Priem, J., & Costello, K. L. (2010). How and why scholars cite on Twitter. Presented at the American Society for Information Science & Technology Annual Meeting, Pittsburgh, PA, 22–27 October 2010.

  • Priem, J., & Hemminger, B. M. (2010). Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web. First Monday, 15(7). Retrieved from http://www.uic.edu/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/2874/2570.

  • Priem, J., Taraborelli, D., Groth, P. & Neylon, C. (2010). Alt-metrics: A manifesto. Retrieved July 25, 2011, from http://altmetrics.org/manifesto/.

  • Research Information Network. (2010). If you build it, will they come? How researchers perceive and use web 2.0 | Research Information Network. Retrieved August 6, 2010, from http://www.rin.ac.uk/our-work/communicating-and-disseminating-research/use-and-relevance-web-20-researchers.

  • Shepherd, P. T. (2007). Final report on the investigation into the feasibility of developing and implementing journal usage factors. Retrieved January 2, 2011, from http://www.uksg.org/sites/uksg.org/files/FinalReportUsageFactorProject.pdf.

  • Smith, A. G. (2011). Wikipedia and institutional repositories: An academic symbiosis? Proceedings of the ISSI 2011 Conference. Presented at the 13th International Conference of the International Society for Scientometrics & Informetrics (pp. 794–800), Durban, 4–7 July 2011.

  • Taraborelli, D. (2008). Soft peer review: Social software and distributed scientific evaluation. In 8th International Conference on the Design of Cooperative Systems. Carry-le-Rouet: Institut d’Etudes Politiques d’Aix-en-Provence, Aix-en-Provence (pp. 99–110), France.

  • Taraborelli, D. (2010). ReaderMeter: Research impact, crowdsourced. Retrieved July 28, 2011, from http://readermeter.org/.

  • Thelwall, M. (2008). Bibliometrics to webometrics. Journal of information science, 34(4), 605–621.

    Article  Google Scholar 

  • Thelwall, M., & Kousha, K. (2008). Online presentations as a source of scientific impact? An analysis of PowerPoint files citing academic journals. Journal of the American Society for Information Science and Technology, 59(5), 805–815.

    Article  Google Scholar 

  • Vaughan, L., & Shaw, D. (2003). Bibliographic and web citations: What is the difference? Journal of the American Society for Information Science and Technology, 54(14), 1313–1322.

    Article  Google Scholar 

  • Vaughan, L., & Shaw, D. (2005). Web citation data for impact assessment: a comparison of four science disciplines. Journal of the American Society for Information Science and Technology, 56(10), 1075–1087.

    Article  Google Scholar 

  • Vaughan, L., & Shaw, D. (2008). A new look at evidence of scholarly citation in citation indexes and from web sources. Scientometrics, 74(2), 317–330. doi:10.1007/s11192-008-0220-2.

    Article  Google Scholar 

  • Ware, M., & Monkman, M. (2008). Peer review in scholarly journals: Perspective of the scholarly community-an international study. Publishing Research Consortium. Retrieved from .

  • Weller, K., & Puschmann, C. (2011). Twitter for scientific communication: How can citations/references be identified and measured? Proceedings of the ACM WebSci’11 (pp. 1–4). Koblenz, Germany, 14–17 June 2011.

  • Zaugg, H., West, R. E., Tateishi, I., & Randall, D. L. (2011). Mendeley: Creating communities of scholarly inquiry through research collaboration. TechTrends, 55(1), 32–36.

    Article  Google Scholar 

Download references

Acknowledgments

An extended version of a paper presented at the 13th International Conference on Scientometrics and Informetrics, Durban (South Africa), 4–7 July 2011 (Li et al. 2011).

Author information

Authors and Affiliations

  1. Peter F. Bronfman Library, York University, 4700 Keele Street, Toronto, ON, M3J 1P3, Canada

    Xuemei Li

  2. Statistical Cybermetrics Research Group, School of Technology, University of Wolverhampton, Wulfruna Street, Wolverhampton, WV1 1LY, UK

    Mike Thelwall

  3. Biomedical Branch Library, Gordon and Leslie Diamond Health Care Centre, University of British Columbia, 2775 Laurel Street, Floor 2, Vancouver, BC V5Z 1M9, Canada

    Dean Giustini

Authors
  1. Xuemei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mike Thelwall
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dean Giustini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xuemei Li.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Li, X., Thelwall, M. & Giustini, D. Validating online reference managers for scholarly impact measurement. Scientometrics 91, 461–471 (2012). https://doi.org/10.1007/s11192-011-0580-x

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-011-0580-x

Keywords

Search

Navigation