Fugang Wang
Geoffrey C. Fox
ABSTRACT
We present a framework that (a) integrates publication and citation data retrieval, (b) allows scientific impact metrics generation at different aggregation levels, and (c) provides correlation analysis of impact metrics based on publication and citation data with resource allocation for a computing facility. Furthermore, we use this framework to conduct a scientific impact metrics evaluation of XSEDE. We carry out an extensive statistical analysis correlating XSEDE allocation size to the impact metrics aggregated by project and field of science. This analysis not only helps to provide an indication of XSEDE's scientific impact, but also provides insight regarding maximizing the return on investment in terms of allocation by taking into account the field of science or project based impact metrics. The findings from this analysis can be utilized by the XSEDE resource allocation committee to help assess and identify projects with higher scientific impact. It can also help provide metrics regarding the return on investment for XSEDE resources, or other institutional or campus resources for which an analysis of impact based on publications is important.
- altmetrics. Web Page. URL: http://altmetrics.org/manifesto/.Google Scholar
- Google Scholar. Web Page. URL: http://scholar.google.com/.Google Scholar
- i-10 index | google scholar citations open to all. URL: http://googlescholar.blogspot.com/2011/11/google-scholar-citations-open-to-all.html.Google Scholar
- ISI Web of Science. Web Page. URL: http://wokinfo.com/.Google Scholar
- Mendeley. Web Page. URL: http://www.mendeley.com/.Google Scholar
- Microsoft Academic Search. Web Page. URL: http://academic.research.microsoft.com/.Google Scholar
- nanoHUB.org - Citations. Web Page. URL: https://nanohub.org/citations.Google Scholar
- Publish or Perish. Web Page. URL: http://www.harzing.com/pop.htm.Google Scholar
- Tas scientific impact metrics and analysis dev/testing portal. URL: http://fgdev.pti.indiana.edu:8088/xdportalpub/.Google Scholar
- XSEDE. Web Page. URL: https://www.xsede.org/.Google Scholar
- J. Bollen, G. Fox, and P. R. Singhal. How and where the TeraGrid supercomputing infrastructure benefits science. Journal of Informetrics, 5(1):114--121, 2011.Google ScholarCross Ref
- J. Bollen, M. A. Rodriguez, and H. Van de Sompel. MESUR: Usage-based Metrics of Scholarly Impact. In Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries, JCDL '07, pages 474--474, New York, NY, USA, 2007. ACM. URL: http://doi.acm.org/10.1145/1255175.1255273, doi:10.1145/1255175.1255273. Google ScholarDigital Library
- J. Bollen, H. Van de Sompel, A. Hagberg, and R. Chute. A principal component analysis of 39 scientific impact measures. PloS one, 4(6):e6022, 2009.Google ScholarCross Ref
- J. Bollen, H. Van de Sompel, and M. A. Rodriguez. Towards Usage-based Impact Metrics: First Results from the Mesur Project. In Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries, JCDL '08, pages 231--240, New York, NY, USA, 2008. ACM. URL: http://doi.acm.org/10.1145/1378889.1378928, doi:10.1145/1378889.1378928. Google ScholarDigital Library
- L. Egghe. Theory and practise of the g-index. Scientometrics, 69(1):131--152, 2006.Google ScholarCross Ref
- T. R. Furlani, B. L. Schneider, M. D. Jones, J. Towns, D. L. Hart, S. M. Gallo, R. L. DeLeon, C.-D. Lu, A. Ghadersohi, R. J. Gentner, A. K. Patra, G. von Laszewski, F. Wang, J. T. Palmer, and N. Simakov. Using xdmod to facilitate xsede operations, planning and analysis. In Proceedings of the Conference on Extreme Science and Engineering Discovery Environment: Gateway to Discovery, XSEDE '13, pages 46:1--46:8, New York, NY, USA, 2013. ACM. URL: http://doi.acm.org/10.1145/2484762.2484763, doi:10.1145/2484762.2484763. Google ScholarDigital Library
- J. E. Hirsch. An index to quantify an individual's scientific research output. Proceedings of the National academy of Sciences of the United States of America, 102(46):16569--16572, 2005.Google ScholarCross Ref
- J. Kaur, D. T. Hoang, X. Sun, L. Possamai, M. JafariAsbagh, S. Patil, and F. Menczer. Scholarometer: A social framework for analyzing impact across disciplines. PloS one, 7(9):e43235, 2012.Google ScholarCross Ref
- P. Thomas and D. Watkins. Institutional research rankings via bibliometric analysis and direct peer review: A comparative case study with policy implications. Scientometrics, 41(3):335--355, 1998.Google ScholarCross Ref
- K. Yang and L. I. Meho. Citation analysis: a comparison of Google Scholar, Scopus, and Web of Science. Proceedings of the American Society for Information Science and Technology, 43(1):1--15, 2006.Google ScholarCross Ref
Index Terms
- Towards a Scientific Impact Measuring Framework for Large Computing Facilities - a Case Study on XSEDE
Recommendations
Comprehensive Evaluation of XSEDE’s Scientific Impact using Semantic Scholar Data
PEARC '21: Practice and Experience in Advanced Research ComputingThe United States science and engineering community faces multiple challenges related to funding and funding policies for science and engineering. A framework is needed to evaluate the impact of scientific facilities and instruments. In this paper, we ...
Using XDMoD to facilitate XSEDE operations, planning and analysis
XSEDE '13: Proceedings of the Conference on Extreme Science and Engineering Discovery Environment: Gateway to DiscoveryThe XDMoD auditing tool provides, for the first time, a comprehensive tool to measure both utilization and performance of high-end cyberinfrastructure (CI), with initial focus on XSEDE. Here, we demonstrate, through several case studies, its utility for ...
Evaluating the Scientific Impact of XSEDE
PEARC '18: Proceedings of the Practice and Experience on Advanced Research ComputingWe use the bibliometrics approach to evaluate the scientific impact of XSEDE. By utilizing publication data from various sources, e.g., ISI Web of Science and Microsoft Academic Graph, we calculate the impact metrics of XSEDE publications and show how ...
Comments