Abstract
Policy document mention is considered to indicate the significance and societal impact of scientific product. However, the accuracy of policy document altmetrics data needs to be evaluated to fully understand its strength and limitation. An in-depth coding analysis was conducted on sample policy documents records of Altmetric.com database. The sample consists of 2079 records from all 79 distinct policy document source platforms tracked by the database. Errors about mentioned publications in the policy documents (type A error) are found in 8% of the records, while errors about either the recorded policy documents or the mentioned publications in the altmetrics database (type B error) are found in 70% of the records. In type B error, policy document link error (5% of the records) could be attributable to the policy document website, transcription error (52% of the records) could be attributable to the third-party bibliographic data provider. These two categories of error are relatively minor and may have limited influence on altmetrics research and practices. False positive policy document mention (13% of the records), however, could be attributable to the Altmetric database and may diminish the validity of research based on the policy document altmetrics data. The underlying reasons remain to be further investigated. Considering the high complexity of extracting mentions of publications from various sources and formats of policy documents as well as its short history, Altmetric database has achieved excellent performance.
Similar content being viewed by others
References
Aduku, K. J., Thelwall, M., & Kousha, K. (2017). Do Mendeley reader counts reflect the scholarly impact of conference papers? An investigation of computer science and engineering. Scientometrics, 112(1), 573–581.
Archambault, É., Campbell, D., Gingras, Y., & Larivière, V. (2009). Comparing bibliometric statistics obtained from the Web of Science and Scopus. Journal of the American Society for Information Science and Technology, 60(7), 1320–1326.
Bar-Ilan, J. (2014). JASIST@ Mendeley revisited. altmetrics14: Expanding impacts and metrics. In Workshop at web science conference 2014.
Bornmann, L., & Haunschild, R. (2015). Which people use which scientific papers? An evaluation of data from F1000 and Mendeley. Journal of Informetrics, 9(3), 477–487.
Bornmann, L., Haunschild, R., & Marx, W. (2016). Policy documents as sources for measuring societal impact: How often is climate change research mentioned in policy-related documents? Scientometrics, 109(3), 1477–1495.
Buchanan, R. A. (2006). Accuracy of cited references: The role of citation databases. College and Research Libraries, 67(4), 292–303.
Calver, M. C., Goldman, B., Hutchings, P. A., & Kingsford, R. T. (2017). Why discrepancies in searching the conservation biology literature matter. Biological Conservation, 213, 19–26.
Chamberlain, S. (2013). Consuming article-level metrics: Observations and lessons. Information Standards Quarterly, 25(2), 4–13.
Chen, X. J. (2014). Study of quality of references and its auditing methods. Chinese Journal of Scientific and Technical Periodicals, 25(9), 1145–1148. (in Chinese).
Donner, P. (2017). Document type assignment accuracy in the journal citation index data of Web of Science. Scientometrics, 113(1), 219–236.
Fang, Z., & Costas, R. (2020). Studying the accumulation velocity of altmetric data tracked by Altmetric.com. Scientometrics, 123, 1077–1101. https://doi.org/10.1007/s11192-020-03405-9.
Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2014). Scientific journal publishers and omitted citations in bibliometric databases: Any relationship? Journal of Informetrics, 8(3), 751–765.
Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2016a). The museum of errors/horrors in Scopus. Journal of Informetrics, 10(1), 174–182.
Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2016b). Empirical analysis and classification of database errors in Scopus and Web of Science. Journal of Informetrics, 10(4), 933–953.
Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2016c). Do Scopus and WoS correct old omitted citations? Scientometrics, 107(2), 321–335.
Garfield, E. (1974). Errors-theirs, ours and yours. In Essays of an information scientist (Philadelphia: ISI Pr., 1977), 2: 80–81. Originally published in Current Contents (June 19, 1974): 5–6.
Harzing, A.-W., & Alakangas, S. (2016). Google Scholar, Scopus and the Web of Science: A longitudinal and cross-disciplinary comparison. Scientometrics, 106(2), 787–804.
Haunschild, R., & Bornmann, L. (2017). How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data. Scientometrics, 110(3), 1209–1216.
Haustein, S. (2016). Grand challenges in altmetrics: Heterogeneity, data quality and dependencies. Scientometrics, 108(1), 413–423.
Haustein, S., Bowman, T. D. & Costas, R. (2015). When is an article actually published? An analysis of online availability, publication, and indexation dates. Retrieved from https://arxiv.org/ftp/arxiv/papers/1505/1505.00796.pdf.
Khazragui, H., & Hudson, J. (2015). Measuring the benefits of university research: Impact and the REF in the UK. Research Evaluation, 24(1), 51–62.
Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105–2125.
Meschede, C., & Siebenlist, T. (2018). Cross-metric compatability and inconsistencies of altmetrics. Scientometrics, 115(1), 283–297.
Moed, H. F. (2002). The impact-factors debate: The ISI’s uses and limits. Nature, 415(6873), 731.
Mohammadi, E., Kwasny, M., & Holmes, K. L. (2018). Academic information on Twitter: A user survey. PLoS ONE, 13(5), e0197265.
Newson, R., King, L., Rychetnik, L., et al. (2018). Looking both ways: A review of methods for assessing research impacts on policy and the policy utilisation of research. Health Research Policy and Systems, 16(1), 54.
NISO. (2016). Outputs of the NISO alternative assessment project. Retrieved from https://groups.niso.org/apps/group_public/download.php/17091/NISO+RP-25-2016+Outputs+of+the+NISO+Alternative+Assessment+Project.pdf.
Ortega, J. L. (2018). Reliability and accuracy of altmetric providers: A comparison among Altmetric.com, PlumX and Crossref Event Data. Scientometrics, 116(3), 2123–2138.
Ortega, J. L. (2019). Blogs and news sources coverage in altmetrics data providers: A comparative analysis by country, language, and subject. Scientometrics. https://doi.org/10.1007/s11192-019-03299-2.
Peters, I., Jobmann, A., Eppelin, A., et al. (2014). Altmetrics for large, multidisciplinary research groups: A case study of the Leibniz Association. Libraries in the Digital Age (LIDA) Proceedings, 13, 1–9.
Prins, A. A., Costas, R., van Leeuwen, T., & Wouters, P. F. (2016). Using Google Scholar in research evaluation of humanities and social science programs: A comparison with Web of Science data. Research Evaluation, 25(3), 264–270.
Su, X. N. (2001). Quality control of data in citation indexes. Journal of Library Science in China, 27(2), 76–78. (in Chinese).
Tattersall, A., & Carroll, C. (2018). What can altmetric.com tell us about policy citations of research? An analysis of Altmetric.com data for research articles from the University of Sheffield. Frontiers in Research Metrics and Analytics, 2(9), 1–11.
Wang, Q., & Waltman, L. (2016). Large-scale analysis of the accuracy of the journal classification systems of Web of Science and Scopus. Journal of Informetrics, 10(2), 347–364.
Wooldridge, J., & King, M. B. (2018). Altmetric scores: An early indicator of research impact. Journal of the Association for Information Science and Technology. https://doi.org/10.1002/asi.24122.
Yu, H. (2017). Context of altmetrics data matters: An investigation of count type and user category. Scientometrics, 111(1), 267–283.
Yu, H., Cao, X., Xiao, T., & Yang, Z. (2019). Accuracy of policy document mentions: The role of altmetrics databases. In Proceedings of ISSI 2019—The 17th international conference on scientometrics and informetrics (pp. 477–488). Italy: Sapienza University.
Zahedi, Z., & Costas, R. (2018). General discussion of data quality challenges in social media metrics: Extensive comparison of four major altmetric data aggregators. PLoS ONE, 13(5), e0197326.
Zahedi, Z., Fenner, M., & Costas, R. (2014). How consistent are altmetrics providers? Study of 1000 PLOS ONE publications using the PLOS ALM, Mendeley and Altmetric.com APIs. altmetrics 14. In Workshop at the Web Science conference, Bloomington, USA.
Zahedi, Z., & Haustein, S. (2018). On the relationships between bibliographic characteristics of scientific documents and citation and Mendeley readership counts: A large-scale analysis of Web of Science publications. Journal of Informetrics, 12(1), 191–202.
Zahedi, Z., Haustein, S., & Bowman, T. (2014). Exploring data quality and retrieval strategies for Mendeley reader counts. In SIG/MET workshop, ASIS&T 2014 annual meeting, Seattle. Retrieved from: www.asis.org/SIG/SIGMET/data/uploads/sigmet2014/zahedi.pdf.
Zhao, Q. M. (2009). Analysis of errors in refereces of scientific journals and the prevention strategies. Editorial Friends, 6, 47–49. (in Chinese).
Acknowledgements
The present study is an extended version of an article (Yu et al. 2019) presented at the 17th International Conference on Scientometrics and Informetrics, Rome (Italy), 2–5 September, 2019. The authors would like to thank Longfei Li and Zihan Yin for helping conduct the coding work. The authors would like to thank Altmetric.com company for providing access to the data and Stacy Konkiel for the useful comments. The research is supported by National Natural Science Foundation of China (NO.71804067), Humanity and Social Science Foundation of Ministry of Education of China (18YJC870023), the Fundamental Research Funds for the Central Universities (No. 30920021203).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Yu, H., Cao, X., Xiao, T. et al. How accurate are policy document mentions? A first look at the role of altmetrics database. Scientometrics 125, 1517–1540 (2020). https://doi.org/10.1007/s11192-020-03558-7
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-020-03558-7