Skip to main content
SpringerLink
Log in

Web Citation Indicators for Wider Impact Assessment of Articles

  • Chapter
Springer Handbook of Science and Technology Indicators

Part of the book series: Springer Handbooks ((SHB))

Abstract

There are many ways in which academic articles can be used outside research contexts for teaching, culture, medical practice, business, policy making, or knowledge communication. Articles that have significant wider benefits may therefore be undervalued if they are assessed through conventional citation indicators and sources (e. g., the Web of Science ( ) and Scopus). A range of online document genres and sources may help to evaluate these broader impacts of articles, including academic syllabi, textbooks, clinical trials or guidelines, patents, encyclopedia articles, and grey literature publications. Web citations can be used as a quantitative impact indicator for monitoring the wider impact of articles, especially in the arts, humanities, and social sciences where many research outputs have value beyond academia. This article reviews literature about the web citation analysis of articles and explains different methods to capture web citations from a range of online sources via commercial search engines. The applications and limitations of web citation analysis for wider impact assessment of articles are discussed, in addition to practical advice for data gathering. New web citation indicators can help research evaluation peer review and citation analysis by giving additional information about the wider benefits of published research when a type of impact (e. g., teaching, commercial, or clinical impact) is required to be assessed by authors, research funders, or evaluators in addition to their academic research impact.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 299.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 379.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • H.F. Moed: Citation Analysis in Research Evaluation (Springer, New York 2005)

    Google Scholar 

  • REF: Decisions on assessing research impact, http://www.ref.ac.uk/media/ref/content/pub/decisionsonassessingresearchimpact/01_11.pdf (2011)

  • T.C. Almind, P. Ingwersen: Informetric analyses on the World Wide Web: Methodological approaches to “Webometrics”, J. Doc. 53(4), 404–426 (1997)

    Article  Google Scholar 

  • B. Cronin: Bibliometrics and beyond: some thoughts on web-based citation analysis, J. Inf. Sci. 27(1), 1–7 (2001)

    Article  Google Scholar 

  • G. Lewison: Beyond outputs: New measures of biomedical research impact, Aslib Proc. 55(1/2), 32–42 (2003)

    Article  Google Scholar 

  • M. Thelwall, L. Vaughan, L. Björneborn: Webometrics, Annu. Rev. Inf. Sci. Technol. 39, 81–135 (2005)

    Article  Google Scholar 

  • J. Bar-Ilan: Informetrics at the beginning of the 21st century-A review, J. Informetr. 2(1), 1–52 (2008)

    Article  Google Scholar 

  • M. Thelwall: Introduction to Webometrics: Quantitative web research for the social sciences. In: Synthesis Lectures on Information Concepts, Retrieval, and Services (Morgan and Claypool Publishers, San Francisco 2009) pp. 1–116

    Google Scholar 

  • L. Vaughan, D. Shaw: Bibliographic and Web citations: What is the difference?, J. Am. Soc. Inf. Sci. Technol. 54(14), 1313–1324 (2003)

    Article  Google Scholar 

  • L. Vaughan, D. Shaw: Web citation data for impact assessment: A comparison of four science disciplines, J. Am. Soc. Inf. Sci. Technol. 56(10), 1075–1087 (2005)

    Article  Google Scholar 

  • D. Hicks: The difficulty of achieving full coverage of international social science literature and the bibliometric consequences, Scientometrics 44(2), 193–215 (1999)

    Article  Google Scholar 

  • A. Nederhof: Bibliometric monitoring of research performance in the social sciences and the humanities: A review, Scientometrics 66(1), 81–100 (2006)

    Article  Google Scholar 

  • K. Kousha, M. Thelwall: Web impact metrics for research assessment. In: Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact, ed. by B. Croninand, C.R. Sugimoto (MIT Press, Cambridge 2014) pp. 289–306

    Google Scholar 

  • The ACUMEN Consortium: Guidelines for Good Evaluation Practice with the ACUMEN Portfolio (2014) http://research-acumen.eu/wp-content/uploads/D6.14-Good-Evaluation-Practices.pdf

  • J. Wilsdon, L. Allen, E. Belfiore, P. Campbell, S. Curry, S. Hill, R. Jones, R. Kain, S. Kerridge, M. Thelwall, J. Tinkler: The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management (Higher Education Funding Council for England, Stoke Gifford 2015)

    Book  Google Scholar 

  • M. Thelwall, K. Kousha, A. Dinsmore, K. Dolby: Alternative metric indicators for funding scheme evaluations, Aslib J. Inf. Manag. 68(1), 2–18 (2016)

    Article  Google Scholar 

  • J. Priem, D. Taraborelli, P. Groth, C. Neylon: Altmetrics: A manifesto, http://altmetrics.org/manifesto/ (2010)

  • J. Bar-Ilan: Which h-index? – A comparison of WoS, Scopus and Google Scholar, Scientometrics 74(2), 257–271 (2008)

    Article  Google Scholar 

  • M. Thelwall, K. Kousha: Web indicators for research evaluation, Part 1: Citations and links to academic articles from the web, Prof. Inf. 24(5), 587–606 (2015)

    Google Scholar 

  • M. Thelwall: Web Indicators for Research Evaluation: A Practical Guide (Morgan & Claypool, San Rafael 2017)

    Google Scholar 

  • P. Wouters, R. Costas: Users, Narcissism and Control: Tracking the Impact of Scholarly Publications in the 21st Century (SURFfoundation, Utrecht 2012) pp. 847–857

    Google Scholar 

  • J. Bar-Ilan: The Web as information source on informetrics? A content analysis, J. Am. Soc. Inf. Sci. 51(5), 432–443 (2000)

    Article  Google Scholar 

  • K. Kousha, M. Thelwall: How is science cited on the Web? A classification of Google unique web citations, J. Am. Soc. Inf. Sci. Technol. 58(11), 1631–1644 (2007)

    Article  Google Scholar 

  • K. Kousha, M. Thelwall: Motivations for URL citations to open access library and information science articles, Scientometrics 68(3), 501–517 (2006)

    Article  Google Scholar 

  • E. Jepsen, P. Seiden, P. Ingwersen, L. Björneborn, P. Borlund: Characteristics of scientific Web publications: Preliminary data gathering and analysis, J. Am. Soc. Inf. Sci. Technol. 55(14), 1239–1249 (2004)

    Article  Google Scholar 

  • K. Kousha: Characteristics of open access web citation networks: A multidisciplinary study, Aslib Proc. 61(4), 394–406 (2009)

    Article  Google Scholar 

  • J. Bar-Ilan: Expectations versus reality – Search engine features needed for Web research at mid 2005, Cybermetrics 9(2) (2005), https://dialnet.unirioja.es/descarga/articulo/1292632.pdf

  • D. Wilkinson, P. Sud, M. Thelwall: Substance without citation: Evaluating the online impact of grey literature, Scientometrics 98(2), 797–806 (2014)

    Article  Google Scholar 

  • M. Thelwall, K. Kousha: Online presentations as a source of scientific impact?: An analysis of PowerPoint files citing academic journals, J. Am. Soc. Inf. Sci. Technol. 59(5), 805–815 (2008)

    Article  Google Scholar 

  • M. Thelwall, K. Kousha: Figshare: A universal repository for academic resource sharing?, Online Inf. Rev. 40(3), 333–346 (2016)

    Article  Google Scholar 

  • M. Thelwall, K. Kousha: SlideShare presentations, citations, users and trends: A professional site with academic and educational uses, J. Assoc. Inf. Sci. Technol. 68(8), 1989–2003 (2017)

    Article  Google Scholar 

  • K. Kousha, M. Thelwall: Assessing the impact of research on teaching: An automatic analysis of online syllabuses in science and social sciences, J. Am. Soc. Inf. Sci. Technol. 59(13), 2060–2069 (2008)

    Article  Google Scholar 

  • K. Kousha, M. Thelwall: An automatic method for assessing the teaching impact of books from online academic syllabi, J. Assoc. Inf. Sci. Technol. 67(12), 2993–3007 (2015)

    Article  Google Scholar 

  • K. Kousha, M. Thelwall: Are Wikipedia citations important evidence of the impact of scholarly articles and books?, J. Assoc. Inf. Sci. Technol. 68(3), 762–779 (2017)

    Article  Google Scholar 

  • J. Priem, H.A. Piwowar, B.M. Hemminger: Altmetrics in the wild: using social media to explore scholarly impact, http://arxiv.org/abs/1203.4745v1 (2012)

  • M.D. Bould, E.S. Hladkowicz, A.E. Pigford, L. Ufholz, T. Postonogova, E. Shin, S. Boet: References that anyone can edit: Review of Wikipedia citations in peer reviewed health science literature, BMJ (Online) 348, g1585 (2014)

    Google Scholar 

  • D. Kryl, L. Allen, K. Dolby, B. Sherbon, I. Viney: Tracking the impact of research on policy and practice: Investigating the feasibility of using citations in clinical guidelines for research evaluation, BMJ Open 2(2), e000897 (2012)

    Article  Google Scholar 

  • M. Thelwall, N. Maflahi: Guideline references and academic citations as evidence of the clinical value of health research, J. Assoc. Inf. Sci. Technol. 67(4), 960–966 (2016)

    Article  Google Scholar 

  • M. Thelwall, K. Kousha: Are citations from clinical trials evidence of higher impact research? An analysis of ClinicalTrials.gov, Scientometrics 109(2), 1341–1351 (2016)

    Article  Google Scholar 

  • M. Thelwall, K. Kousha, M. Abdoli: Is medical research informing professional practice more highly cited? Evidence from AHFS DI Essentials in Drugs.com, Scientometrics 112(1), 509–527 (2017)

    Article  Google Scholar 

  • Australian Government: National Health and Medical Research Council, https://www.nhmrc.gov.au

  • Canadian Medical Association: CPG Infobase: Clinical Practice Guidelines, https://www.cma.ca/En/Pages/clinical-practice-guidelines.aspx

  • Scottish Intercollegiate Guidelines Network: Current Guidelines, https://www.sign.ac.uk/our-guidelines.html

  • R. Haunschild, L. Bornmann: How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data, Scientometrics 110(3), 1209–1216 (2017)

    Article  Google Scholar 

  • M. Thelwall, K. Kousha: ResearchGate versus Google Scholar: Which finds more early citations?, Scientometrics 112(2), 1125–1131 (2017)

    Article  Google Scholar 

  • I. Mewburn, P. Thomson: Why do academics blog? An analysis of audiences, purposes and challenges, Stud. High. Educ. 38(8), 1105–1119 (2013)

    Article  Google Scholar 

  • H. Shema, J. Bar-Ilan, M. Thelwall: Research blogs and the discussion of scholarly information, PLoS ONE 7(5), e35869 (2012)

    Article  Google Scholar 

  • M. Thelwall, S. Haustein, V. Larivière, C. Sugimoto: Do altmetrics work? Twitter and ten other candidates, PLoS ONE 8(5), e64841 (2013)

    Article  Google Scholar 

  • H. Shema, J. Bar-Ilan, M. Thelwall: Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics, J. Assoc. Inf. Sci. Technol. 65(5), 1018–1027 (2014)

    Article  Google Scholar 

  • L.I. Meho, K. Yang: Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar, J. Am. Soc. Inf. Sci. Technol. 58(13), 2105–2125 (2007)

    Article  Google Scholar 

  • J.C.F. de Winter, A.A. Zadpoor, D. Dodou: The expansion of Google Scholar versus Web of Science: A longitudinal study, Scientometrics 98(2), 1547–1565 (2014)

    Article  Google Scholar 

  • H.F. Moed, J. Bar-Ilan, G. Halevi: A new methodology for comparing Google Scholar and Scopus, J. Informetr. 10(2), 533–551 (2016)

    Article  Google Scholar 

  • J. Beel, B. Gipp: Academic search engine spam and Google Scholar's resilience against it, J. Electron. Publ. (2010), https://doi.org/10.3998/3336451.0013.305

    Article  Google Scholar 

  • E.D. López-Cózar, N. Robinson-García, D. Torres-Salinas: The Google Scholar experiment: How to index false papers and manipulate bibliometric indicators, J. Assoc. Inf. Sci. Technol. 65(3), 446–454 (2014)

    Article  Google Scholar 

  • M. Khabsa, C.L. Giles: The number of scholarly documents on the public web, PLoS ONE 9(5), e93949 (2014)

    Article  Google Scholar 

  • A.W.K. Harzing, R. van der Wal: Google Scholar as a new source for citation analysis, Ethics Sci. Environ. Politics 8(1), 61–73 (2008)

    Article  Google Scholar 

  • B. Cronin, H. Snyder, H. Atkins: Comparative citation rankings of authors in monographic and journal literature: A study of sociology, J. Doc. 53(3), 263–273 (1997)

    Article  Google Scholar 

  • J. Gorraiz, P.J. Purnell, W. Glänzel: Opportunities for and limitations of the book citation index, J. Am. Soc. Inf. Sci. Technol. 64(7), 1388–1398 (2013)

    Article  Google Scholar 

  • D. Torres-Salinas, R. Rodríguez-Sánchez, N. Robinson-García, J. Fdez-Valdivia, J.A. García: Mapping citation patterns of book chapters in the Book Citation Index, J. Informetr. 7(2), 412–424 (2013)

    Article  Google Scholar 

  • K. Kousha, M. Thelwall: Google book search: Citation analysis for social science and the humanities, J. Am. Soc. Inf. Sci. Technol. 60(8), 1537–1549 (2009)

    Article  Google Scholar 

  • M. Huang, W. Huang, D. Chen: Technological impact factor: An indicator to measure the impact of academic publications on practical innovation, J. Informetr. 8(1), 241–251 (2014)

    Article  Google Scholar 

  • Y. Liaw, T. Chan, C. Fan, C. Chiang: Can the technological impact of academic journals be evaluated? The practice of non-patent reference (NPR) analysis, Scientometrics 101(1), 17–37 (2014)

    Article  Google Scholar 

  • K. Kousha, M. Thelwall: Patent citation analysis with Google, J. Assoc. Inf. Sci. Technol. 68(1), 48–61 (2017)

    Article  Google Scholar 

  • S.E. Hug, M. Ochsner, M.P. Brändle: Citation analysis with Microsoft Academic, Scientometrics 111(1), 371–378 (2017)

    Article  Google Scholar 

  • A.W. Harzing, S. Alakangas: Microsoft Academic: Is the phoenix getting wings?, Scientometrics 110(1), 371–383 (2017)

    Article  Google Scholar 

  • A.W. Harzing, S. Alakangas: Microsoft Academic is one year old: The Phoenix is ready to leave the nest, Scientometrics 112(3), 1887–1894 (2017)

    Article  Google Scholar 

  • M. Thelwall: Microsoft Academic: A multidisciplinary comparison of citation counts with Scopus and Mendeley for 29 journals, J. Informetr. 11(4), 1201–1212 (2017)

    Article  Google Scholar 

  • M. Thelwall: Does Microsoft Academic find early citations?, Scientometrics 114(1), 325–334 (2018)

    Article  Google Scholar 

  • K. Kousha, M. Thelwall: Can Microsoft Academic assess the early citation impact of in press articles? A multi-discipline exploratory analysis, J. Informetr. 12(1), 287–298 (2018)

    Article  Google Scholar 

  • M. Thelwall: Can Microsoft Academic be used for citation analysis of preprint archives?, The case of the Social Science Research Network, Scientometrics 115(2), 913–928 (2018)

    Article  Google Scholar 

  • F.A. Nielsen: Scientific citations in Wikipedia, First Monday (2007), http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/1997/1872

  • M.F. Weigold: Communicating science: A review of the literature, Sci. Commun. 23(2), 164–193 (2001)

    Article  Google Scholar 

  • A. Dudo: Scientists, the media, and the public communication of science, Sociol. Compass 9(9), 761–775 (2015)

    Article  Google Scholar 

  • V. Entwistle: Reporting research in medical journals and newspapers, BMJ 310(6984), 920 (1995)

    Article  Google Scholar 

  • C. Bartlett, J. Sterne, M. Egger: What is newsworthy? Longitudinal study of the reporting of medical research in two British newspapers, Br. Med. J. 325(7355), 81–84 (2002)

    Article  Google Scholar 

  • G. Lewison, S. Tootell, P. Roe, R. Sullivan: How do the media report cancer research? A study of the UK's BBC website, Br. J. Cancer 99(4), 569 (2008)

    Article  Google Scholar 

  • K. Kousha, M. Thelwall: An automatic method to identify citations to journals in news stories: A case study of the UK newspapers citing Web of Science journals, Scientometrics (submitted)

    Google Scholar 

  • K. Kousha, M. Thelwall: An automatic method for extracting citations from Google Books, J. Assoc. Inf. Sci. Technol. 66(2), 309–320 (2015)

    Article  Google Scholar 

  • K. Kousha, M. Thelwall: Google Scholar citations and Google Web/URL citations: A multi-discipline exploratory analysis, J. Am. Soc. Inf. Sci. Technol. 58(7), 1055–1065 (2007)

    Article  Google Scholar 

  • A.V. Kulkarni, B. Aziz, I. Shams, J.W. Busse: Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals, J. Am. Med. Assoc. 302(10), 1092–1096 (2009)

    Article  Google Scholar 

  • J. Mingers, E.A.E.C.G. Lipitakis: Counting the citations: A comparison of Web of Science and Google Scholar in the field of business and management, Scientometrics 85(2), 613–625 (2010)

    Article  Google Scholar 

  • E.M. Lasda Bergman: Finding citations to social work literature: The relative benefits of using Web of Science, Scopus, or Google Scholar, J. Acad. Librar. 38(6), 370–379 (2012)

    Article  Google Scholar 

  • J. Bar-Ilan: Data collection methods on the web for informetric purposes – A review and analysis, Scientometrics 50(1), 7–32 (2001)

    Article  Google Scholar 

  • S. Lawrence, C.L. Giles: Accessibility of information on the web, Nature 400(6740), 107–109 (1999)

    Article  Google Scholar 

  • W. Mettrop, P. Nieuwenhuysen: Internet search engines – fluctuations in document accessibility, J. Doc. 57(5), 623–651 (2001)

    Article  Google Scholar 

  • D. Lewandowski: Evaluating the retrieval effectiveness of web search engines using a representative query sample, J. Assoc. Inf. Sci. Technol. 66(9), 1763–1775 (2015)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Statistical Cybermetrics Research Group, University of Wolverhampton, Wolverhampton, UK

    Kayvan Kousha

Authors
  1. Kayvan Kousha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kayvan Kousha .

Editor information

Editors and Affiliations

  1. ECOOM and Faculty of Economics and Business, KU Leuven, Leuven, Belgium

    Wolfgang Glänzel

  2. Amsterdam, The Netherlands

    Henk F. Moed

  3. Competence Center Policy – Industry – Innovation, Fraunhofer Institute for Systems and Innovation Research ISI, Karlsruhe, Germany

    Ulrich Schmoch

  4. Faculty of Science and Engineering, University of Wolverhampton, Wolverhampton, UK

    Mike Thelwall

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer International Publishing AG, part of Springer Nature

About this chapter

Cite this chapter

Kousha, K. (2019). Web Citation Indicators for Wider Impact Assessment of Articles. In: Glänzel, W., Moed, H.F., Schmoch, U., Thelwall, M. (eds) Springer Handbook of Science and Technology Indicators. Springer Handbooks. Springer, Cham. https://doi.org/10.1007/978-3-030-02511-3_31

Download citation

Publish with us

Policies and ethics

Search

Navigation