Craig A. Stewart
Alan Sill
ABSTRACT
In recent years, considerable attention has been given to assessing the value of investments in cyberinfrastructure (CI). This paper includes a survey of current methods for the assessment of financial returns on investment (ROI) in CI. Applying the financial concept of ROI proves challenging with regard to a service that, in most academic environments, does not generate a "sold amount" such as one would find in the buying and selling of stocks. The paper concludes with a discussion of future research directions and challenges in the assessment of financial ROI in CI. This work is intended less as a definitive guide than as a starting point for further exploration in the assessment of CI's value for scientific research.
- {n. d.}. Advanced CyberInfrastructure - Research and Education Facilitators. https://aciref.org/Google Scholar
- {n. d.}. Aristotle Cloud Federation: Federated Cloud Model. http://federatedcloud.orgGoogle Scholar
- {n. d.}. Campus Research Computing Consortium. https://carcc.orgGoogle Scholar
- {n. d.}. Cornell University Center for Advanced Computing. https://www.cac.cornell.edu/Google Scholar
- {n. d.}. Economic Impact Analysis for Planning. http://www.implan.com/Google Scholar
- {n. d.}. Fermilab HEPCloud. https://hepcloud.fnal.gov/Google Scholar
- {n. d.}. High Performance Computing (HPC) Research. https://hyperionresearch.com/Google Scholar
- {n. d.}. ROI with HPC. https://www.hpcuserforum.com/ROI/Google Scholar
- {n. d.}. The CMS Experiment. https://cms.cern/Google Scholar
- {n. d.}. XSEDE. https://www.xsede.org/Google Scholar
- {n. d.}. XSEDE Campus Champions. https://www.xsede.org/web/site/community-engagement/campus-championsGoogle Scholar
- 2004. W.K. Kellog Foundation Logic Model Development Guide.Google Scholar
- 2007. Sourcebook for Evaluating Global and Regional Partnership Programs: Indicative Principles and Standards. Washington, D.C. http://www.oecd.org/development/evaluation/dcdndep/37981082.pdfGoogle Scholar
- 2018. Colleges With the Greatest Percentage Decreases in Full-Time Undergraduates. Chronicle of Higher Education 65, 15 (12 2018). https://www.chronicle.com/article/Colleges-With-the-Greatest/245285Google Scholar
- Amy Apon, Stanley Ahalt, Vijay Dantuluri, Constantin Gurdgiev, Moez Limayem, Linh Ngo, and Michael Stealey. 2010. High Performance Computing Instrumentation and Research Productivity in U.S. Universities. Journal of Information Technology Impact 10, 2 (9 2010), 87--98. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1679248Google Scholar
- Jill Barshay. 2018. College students predicted to fall by more than 15% after the year 2025. The Hechinger Report (9 2018). https://hechingerreport.org/college-students-predicted-to-fall-by-more-than-15-after-the-year-2025/Google Scholar
- L. Bauerdick, B. Bockelman, D. Dykstra, S. Fuess, G. Garzoglio, O. Gutsche, B. Holzman, D. Hufnagel, Hanjin Kim, R. Kennedy, D.I. Mason, P. Spentzouris, S. Timm, A. Tiradani, and E. Vaandering. 2017. Experience in using commercial clouds in CMS. Journal of Physics Conference Series 898, 5 (2017).Google ScholarCross Ref
- Evan F. Bollig and James C. Wilgenbusch. 2018. From Bare Metal to Virtual. In PEARC '18 Proceedings of the Practice and Experience on Advanced Research Computing. ACM, New York, NY, USA, 13:1--13:8.Google Scholar
- Sandy Dall'erba and Zhangliang Chen. 2017. The Impact of Blue Waters on the Economy of Illinois. Technical Report. 4 pages. http://www.ncsa.illinois.edu/assets/pdf/about/bw_economic_impact.pdfGoogle Scholar
- Robert L DeLeon, Thomas R Furlani, Steven M Gallo, Joseph P White, Matthew D Jones, Abani Patra, Martins Innus, Thomas Yearke, Jeffrey T Palmer, Jeanette M Sperhac, Ryan Rathsam, Nikolay Simakov, Gregor von Laszewski, and Fugang Wang. 2015. TAS View of XSEDE Users and Usage. In Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure (XSEDE '15). ACM, New York, NY, USA, 21:1--21:8. Google ScholarDigital Library
- Lizanne DeStefano and Lorna Rivera. 2015. Cyberinfrastructure Value Assessment Report. Technical Report. http://hdl.handle.net/2022/20568Google Scholar
- Robert Henschel, Phillip M. Nista, Matthias Lieber, Brian J. Haas, Le-Shin Wu, and Richard D. LeDuc. 2012. Trinity RNA-Seq assembler performance optimization. (2012), 1. Google ScholarDigital Library
- Will Kenton. 2018. Investopedia: Cost Center. https://www.investopedia.com/terms/c/cost-center.aspGoogle Scholar
- Michael R. Kinney and Cecily A. Raiborn. 2012. Cost Accounting - Foundations and Evolutionse (9th ed.).Google Scholar
- D.L. Kirkpatrick. 1998. Evaluating Training Programs: The Four Levels (2nd ed.). BerrettKoehler Publishers, San Francisco, CA.Google Scholar
- D. Lifka, R. Reynolds, Susan Mehringer, and P. Redfern. {n. d.}. Overview of the Cornell University Center for Advanced Computing Sustainable Funding Model. In NSF Workshop on Sustainable Funding and Business Models for High Performance Computing Centers. Ithaca, NY.Google Scholar
- National Center for Supercomputing Applications. 2017. NCSA's Blue Waters project provides $1.08 billion direct return to Illinois' economy. http://www.ncsa.illinois.edu/news/story/ncsas_blue_waters_project_provides_1.08_billion_direct_return_to_illinois_eGoogle Scholar
- Jeffrey T. Palmer, M. Gallo, Steven, Thomas R. Furlani, Matthew D.Jones, Robert L. DeLeon, Joseph P. White, Nikolay Simakov, Abani K. Patra, Jeanette Sperhac, Thomas Yearke, Ryan Rathsam, Martins Innus, Cynthia Cornelius, James Browne, William L. Barth, and T. Evans, Richard. 2015. Open XDMoD: A Tool for the Comprehensive Management of High Performance Computing Resources. Computing in Science and Engineering 17, 4 (2015), 52--62.Google ScholarDigital Library
- HHS Office of Civil Rights. 2016. Widespread HIPAA vulnerabilities result in $2.7 million settlement with Oregon Health & Science University {Press release}. https://wayback.archive-it.org/3926/20170128024121/https://www.hhs.gov/about/news/2016/07/18/widespread-hipaa-vulnerabilities-result-in-settlement-with-oregon-health-science-university.htmlGoogle Scholar
- Olga Scrivener, Gangadeep Singh, Sara E. Bouchard, Scott C. Hutcheson, Ben Fulton, Matthew R. Link, and Katy Borner. 2018. XD Metrics on Demand Value Analytics: Visualizing the Impact of Internal Information Technology Investments on External Funding, Publications, and Collaboration Networks. Frontiers in Research Metrics and Analytics (2018).Google Scholar
- Troy Segal. 2019. Investopedia: Profit Center. https://www.investopedia.com/terms/p/profitcentre.aspGoogle Scholar
- P.E. Stephan. 2012. How Economics Shapes Science, Vol. 1. Harvard University Press, Cambridge, MA.Google Scholar
- Craig Stewart, David Hancock, Amy Apon, Alan Sill, D.F. McMullen, Thomas Furlani, Julie Wernert, Nicholas Berente, Ron Payne, David Lifka, and Thomas Cheatham. 2019. Assessment of return on investment in cyberinfrastructure facilities: A survey of current methods for assessing non-financial impact. In Workshop Proceedings: Humans in the Loop: Enabling and Facilitating Research on Cloud Computing. PEARC19. Chicago, IL. Google ScholarDigital Library
- Craig Stewart, Beth Plale, Von Welch, Marlon Pierce, Geoffrey C. Fox, Thomas G. Doak, David Y. Hancock, Robert Henschel, Matthew R. Link, Therese Miller, Eric Wernert, Michael J. Boyles, Ben Fulton, Le Mai Weakley, Robert Ping, Tassie Gniady, and Winona Snapp-Childs. 2017. Pervasive Technology Institute Annual Report: Research Innovations and Advanced Cyberinfrastructure Services in Support of IU Strategic Goals During FY 2017. Technical Report. Indiana University, Bloomington. https://scholarworks.iu.edu/dspace/handle/2022/21809Google Scholar
- Craig A Stewart, T M Cockerill, I Foster, D Hancock, N Merchant, E Skidmore, D Stanzione, J Taylor, S Tuecke, G Turner, M Vaughn, and N Gaffney. 2015. Jetstream - A self-provisioned, scalable science and engineering cloud environment. In Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure. St. Louis, MO, 29:1--29:8. Google ScholarDigital Library
- Craig A. Stewart, David Y. Hancock, Julie Wernert, Matthew R. Link, Nancy Wilkins-Diehr, Therese Miller, Kelly Gaither, and Winona Snapp-Childs. 2018. Return on Investment for Three Cyberinfrastructure Facilities: A Local Campus Supercomputer, the NSF-Funded Jetstream Cloud System, and XSEDE (the eXtreme Science and Engineering Discovery Environment). In 2018 IEEE/ACM 11th International Conference on Utility and Cloud Computing (UCC). IEEE, 223--236.Google Scholar
- Craig A. Stewart, R. Knepper, M.R. Link, M. Pierce, E.A. Wernert, and N. Wilkins-Diehr. 2017. Cyberinfrastructure, Cloud Computing, Science Gateways, Visualization, and Cyberinfrastructure Ease of Use. In Encyclopedia of Information Science and Technology, Fourth Edition (fourth ed.), M. Khosrow-Pour (Ed.). IGI Global, Hershey, PA, 1063--1074.Google Scholar
- Craig A Stewart, Ralph Roskies, Richard Knepper, Richard L Moore, Justin Whitt, and Timothy M Cockerill. 2015. XSEDE Value Added, Cost Avoidance, and Return on Investment. In Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure (XSEDE '15). ACM, New York, NY, USA, 23:1--23:8. Google ScholarDigital Library
- Craig A. Stewart, Julie A. Wernert, Nancy Wilkins-ÂŋDiehr, and Kelly Gaither. 2018. XSEDE Return on Investment Data and Analysis, 2015--2017. Technical Report. Indiana University, Bloomington Indiana. http://hdl.handle.net/2022/22491Google Scholar
- Abhinav S Thota, Ben Fulton, Le Mai Weakley Weakley, Robert Henschel, David Y Hancock, Matt Allen, Jenett Tillotson, Matt Link, and Craig A Stewart. 2016. A PetaFLOPS Supercomputer As a Campus Resource: Innovation, Impact, and Models for Locally-Owned High Performance Computing at Research Colleges and Universities. In Proceedings of the 2016 ACM on SIGUCCS Annual Conference (SIGUCCS '16). ACM, New York, NY, USA, 61--68. Google ScholarDigital Library
- G von Laszewski, F Wang, G C Fox, D L Hart, T R Furlani, R L DeLeon, and S M Gallo. 2015. Peer Comparison of XSEDE and NCAR Publication Data. In 2015 IEEE International Conference on Cluster Computing. 531--532. Google ScholarDigital Library
Index Terms
- Assessment of financial returns on investments in cyberinfrastructure facilities: A survey of current methods
Recommendations
Return on Investment in Research Cyberinfrastructure: State of the Art
PEARC '22: Practice and Experience in Advanced Research Computing“What is the Return On Investment (ROI) for a cyberinfrastructure system or service?” seems like a natural question to ask. Existing literature shows strong evidence of good return on investment in cyberinfrastructure. This paper summarizes key points ...
XSEDE value added, cost avoidance, and return on investment
XSEDE '15: Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced CyberinfrastructureIt is difficult for large research facilities to quantify a return on the investments that fund their operations. This is because there can be a time lag of years or decades between an innovation or discovery and the realization of its value through ...
Assessment of non-financial returns on cyberinfrastructure: A survey of current methods
HARC '19: Proceedings of the Humans in the Loop: Enabling and Facilitating Research on Cloud ComputingIn recent years, considerable attention has been given to assessing the value of investments in cyberinfrastructure (CI). This paper focuses on assessment of value measured in ways other than financial benefits - what might well be termed impact or ...
Comments