Abstract
In an ideal clinical Natural Language Processing (NLP) ecosystem, researchers and developers would be able to collaborate with others, undertake validation of NLP systems, components, and related resources, and disseminate them. We captured requirements and formative evaluation data from the Veterans Affairs (VA) Clinical NLP Ecosystem stakeholders using semi-structured interviews and meeting discussions. We developed a coding rubric to code interviews. We assessed inter-coder reliability using percent agreement and the kappa statistic. We undertook 15 interviews and held two workshop discussions. The main areas of requirements related to; design and functionality, resources, and information. Stakeholders also confirmed the vision of the second generation of the Ecosystem and recommendations included; adding mechanisms to better understand terms, measuring collaboration to demonstrate value, and datasets/tools to navigate spelling errors with consumer language, among others. Stakeholders also recommended capability to: communicate with developers working on the next version of the VA electronic health record (VistA Evolution), provide a mechanism to automatically monitor download of tools and to automatically provide a summary of the downloads to Ecosystem contributors and funders. After three rounds of coding and discussion, we determined the percent agreement of two coders to be 97.2% and the kappa to be 0.7851. The vision of the VA Clinical NLP Ecosystem met stakeholder needs. Interviews and discussion provided key requirements that inform the design of the VA Clinical NLP Ecosystem.



Similar content being viewed by others
References
I2b2.org. i2b2: informatics for integrating biology & the bedside. 2016. https://www.i2b2.org/. Accessed 4 Jan 2016.
Murphy, S., and Wilcox, A., Mission and sustainability of informatics for integrating biology and the bedside (i2b2). Egems (Generating Evidence & Methods To Improve Patient Outcomes) 2(2), 2014.
Zeng, Q. T., Redd, D., and Divita, G., Characterizing clinical text and sublanguage: A case study of the VA clinical notes. J Health MedInform S3:001, 2011.
Collaboration between VINCI and CHIR. 2016. Hsrd.research.va.gov. http://www.hsrd.research.va.gov/for_researchers/vinci/chir.cfm. Accessed 28 April 2015.
VA Informatics and Computing Infrastructure (VINCI). 2009. Hsrd.research.va.gov. http://www.hsrd.research.va.gov/for_researchers/vinci/. Accessed 1 June 2011.
Welcome to eMerge. 2016. Emerge.mc.vanderbilt.edu. http://emerge.mc.vanderbilt.edu/about-emerge/. Accessed 15 Jan 2016
Open Health Natural Language Processing (OHNLP) Consortium. 2016. Ohnlp.org. http://www.ohnlp.org/index.php/Main_Page. Accessed 16 May 2016.
Ohno-Machado, L., et al., iDASH: Integrating data for analysis, anonymization, and sharing. J. Am. Med. Inform. Assoc. 19(2):196–201, 2012.
Rea, S., et al., Building a robust, scalable and standards-driven infrastructure for secondary use of EHR data: The SHARPn project. J. Biomed. Inform. 45(4):763–771, 2012.
Faems, D., Open innovation: Researching a new paradigm—by H. Chesbrough, W. Vanhaverbeke and J. West. Creat Innov Manag 17(4):334–335, 2008.
Saxenian, A., Regional advantage. Harvard University Press, Cambridge, 1994.
HL7 Clinical Document Architecture. http://www.hl7.org/implement/standards/product_brief.cfm?product_id=7. Accessed 12 Aug 2016.
Lane, N., Miluzzo, E., Lu, H., Peebles, D., Choudhury, T., and Campbell, A., A survey of mobile phone sensing. IEEE Commun. Mag. 48(9):140–150, 2010.
Messerschmitt, D., and Szyperski, C., Software ecosystem: understanding an indispensable technology and industry. MIT Press, 2003.
Brandenburger, A., and Nalebuff, B., Co-opetition. Doubleday, New York, 1996.
Lam, T., Gotz, A., Leveraging the eclipse ecosystem for the scientific community. 2005. https://accelconf.web.cern.ch/accelconf/ica05/proceedings/pdf/O4_009.pdf. Accessed 2 Oct 2015.
Iansiti, M., and Levien, R., The keystone advantage. Harvard Business School Press, Boston, 2004.
Gawer, A. The organization of technological platforms. Technology and organization: essays in honour of Joan Woodward, 287–296, 2010.
Herbsleb, J. D., Muller-Birn, C., and Towne, W. B., The VistA ecosystem: current status and future directions. Carnegie Mellon University, Pittsburgh, 2010.
Wallace, C., and Legro, M., Using formative evaluation in an implementation project to increase vaccination rates in high-risk veterans: QUERI series. Implement. Sci. 3(1):22, 2008.
Flick, U., The SAGE qualitative research kit. Eight volume set. Sage Publications, London, 2007.
Schiller, C., Winters, M., Hanson, H., and Ashe, M., A framework for stakeholder identification in concept mapping and health research: A novel process and its application to older adult mobility and the built environment. BMC Public Health 13(1):428, 2013.
Sittig, D., and Singh, H., A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 19(Suppl 3):i68–i74, 2010.
Guest, G., MacQueen, K., and Namey, E., Applied thematic analysis. Sage Publications, Los Angeles, 2012.
Braun, V., and Clarke, V., Using thematic analysis in psychology. Qual Res Psychol 3(2):77–101, 2006.
McHugh, M. L., Interrater reliability: The kappa statistic. Biochem Med 22(3):276–282, 2012.
Lang, J. P., Redmine. 2006. http://www.redmine.org. Accessed 8 Sept 2016.
Department of Veterans Affairs Blueprint for Excellence, VHA Blueprint for Excellence. 2014. http://www.va.gov/HEALTH/docs/VHA_Blueprint_for_Excellence.pdf. Accessed 12 July 2016.
Acknowledgements
The views expressed are those of the authors and do not necessarily reflect those of the Department of Veterans Affairs, the United States Government, or the academic affiliate organizations. This work was supported by the VA IDEAS 2.0 HSRD Research Center and the CREATE: A VHA NLP Software Ecosystem for Collaborative Development and Integration #CRE 12–315.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethical statement
This study was approved by the VA Central IRB #CIRB 13–17 Zeng.
Funding
This work was supported by project number CRE 12–315 from the United States (U.S) Department of Veterans Affairs VA IDEAS 2.0 HSR&D Research Center. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs, their academic affiliates, or the U.S. Government.
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the University of Utah IRB and VA SLC Research and Development Office and with the 1964 Helsinski declaration and its later amendments or comparable ethical standards.
Informed consent
We obtained informed consent from all interview participants included in the study.
Additional information
This article is part of the Topical Collection on Systems-Level Quality Improvement
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Garvin, J.H., Kalsy, M., Brandt, C. et al. An Evolving Ecosystem for Natural Language Processing in Department of Veterans Affairs. J Med Syst 41, 32 (2017). https://doi.org/10.1007/s10916-016-0681-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10916-016-0681-4