Skip to main content
SpringerLink
Log in

An Evolving Ecosystem for Natural Language Processing in Department of Veterans Affairs

  • Systems-Level Quality Improvement
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

In an ideal clinical Natural Language Processing (NLP) ecosystem, researchers and developers would be able to collaborate with others, undertake validation of NLP systems, components, and related resources, and disseminate them. We captured requirements and formative evaluation data from the Veterans Affairs (VA) Clinical NLP Ecosystem stakeholders using semi-structured interviews and meeting discussions. We developed a coding rubric to code interviews. We assessed inter-coder reliability using percent agreement and the kappa statistic. We undertook 15 interviews and held two workshop discussions. The main areas of requirements related to; design and functionality, resources, and information. Stakeholders also confirmed the vision of the second generation of the Ecosystem and recommendations included; adding mechanisms to better understand terms, measuring collaboration to demonstrate value, and datasets/tools to navigate spelling errors with consumer language, among others. Stakeholders also recommended capability to: communicate with developers working on the next version of the VA electronic health record (VistA Evolution), provide a mechanism to automatically monitor download of tools and to automatically provide a summary of the downloads to Ecosystem contributors and funders. After three rounds of coding and discussion, we determined the percent agreement of two coders to be 97.2% and the kappa to be 0.7851. The vision of the VA Clinical NLP Ecosystem met stakeholder needs. Interviews and discussion provided key requirements that inform the design of the VA Clinical NLP Ecosystem.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. I2b2.org. i2b2: informatics for integrating biology & the bedside. 2016. https://www.i2b2.org/. Accessed 4 Jan 2016.

  2. Murphy, S., and Wilcox, A., Mission and sustainability of informatics for integrating biology and the bedside (i2b2). Egems (Generating Evidence & Methods To Improve Patient Outcomes) 2(2), 2014.

  3. Zeng, Q. T., Redd, D., and Divita, G., Characterizing clinical text and sublanguage: A case study of the VA clinical notes. J Health MedInform S3:001, 2011.

    Google Scholar 

  4. Collaboration between VINCI and CHIR. 2016. Hsrd.research.va.gov. http://www.hsrd.research.va.gov/for_researchers/vinci/chir.cfm. Accessed 28 April 2015.

  5. VA Informatics and Computing Infrastructure (VINCI). 2009. Hsrd.research.va.gov. http://www.hsrd.research.va.gov/for_researchers/vinci/. Accessed 1 June 2011.

  6. Welcome to eMerge. 2016. Emerge.mc.vanderbilt.edu. http://emerge.mc.vanderbilt.edu/about-emerge/. Accessed 15 Jan 2016

  7. Open Health Natural Language Processing (OHNLP) Consortium. 2016. Ohnlp.org. http://www.ohnlp.org/index.php/Main_Page. Accessed 16 May 2016.

  8. Ohno-Machado, L., et al., iDASH: Integrating data for analysis, anonymization, and sharing. J. Am. Med. Inform. Assoc. 19(2):196–201, 2012.

    Article  PubMed  Google Scholar 

  9. Rea, S., et al., Building a robust, scalable and standards-driven infrastructure for secondary use of EHR data: The SHARPn project. J. Biomed. Inform. 45(4):763–771, 2012.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Faems, D., Open innovation: Researching a new paradigm—by H. Chesbrough, W. Vanhaverbeke and J. West. Creat Innov Manag 17(4):334–335, 2008.

    Article  Google Scholar 

  11. Saxenian, A., Regional advantage. Harvard University Press, Cambridge, 1994.

    Google Scholar 

  12. HL7 Clinical Document Architecture. http://www.hl7.org/implement/standards/product_brief.cfm?product_id=7. Accessed 12 Aug 2016.

  13. Lane, N., Miluzzo, E., Lu, H., Peebles, D., Choudhury, T., and Campbell, A., A survey of mobile phone sensing. IEEE Commun. Mag. 48(9):140–150, 2010.

    Article  Google Scholar 

  14. Messerschmitt, D., and Szyperski, C., Software ecosystem: understanding an indispensable technology and industry. MIT Press, 2003.

  15. Brandenburger, A., and Nalebuff, B., Co-opetition. Doubleday, New York, 1996.

    Google Scholar 

  16. Lam, T., Gotz, A., Leveraging the eclipse ecosystem for the scientific community. 2005. https://accelconf.web.cern.ch/accelconf/ica05/proceedings/pdf/O4_009.pdf. Accessed 2 Oct 2015.

  17. Iansiti, M., and Levien, R., The keystone advantage. Harvard Business School Press, Boston, 2004.

    Google Scholar 

  18. Gawer, A. The organization of technological platforms. Technology and organization: essays in honour of Joan Woodward, 287–296, 2010.

  19. Herbsleb, J. D., Muller-Birn, C., and Towne, W. B., The VistA ecosystem: current status and future directions. Carnegie Mellon University, Pittsburgh, 2010.

    Google Scholar 

  20. Wallace, C., and Legro, M., Using formative evaluation in an implementation project to increase vaccination rates in high-risk veterans: QUERI series. Implement. Sci. 3(1):22, 2008.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Flick, U., The SAGE qualitative research kit. Eight volume set. Sage Publications, London, 2007.

    Google Scholar 

  22. Schiller, C., Winters, M., Hanson, H., and Ashe, M., A framework for stakeholder identification in concept mapping and health research: A novel process and its application to older adult mobility and the built environment. BMC Public Health 13(1):428, 2013.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Sittig, D., and Singh, H., A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 19(Suppl 3):i68–i74, 2010.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Guest, G., MacQueen, K., and Namey, E., Applied thematic analysis. Sage Publications, Los Angeles, 2012.

    Book  Google Scholar 

  25. Braun, V., and Clarke, V., Using thematic analysis in psychology. Qual Res Psychol 3(2):77–101, 2006.

    Article  Google Scholar 

  26. McHugh, M. L., Interrater reliability: The kappa statistic. Biochem Med 22(3):276–282, 2012.

    Article  Google Scholar 

  27. Lang, J. P., Redmine. 2006. http://www.redmine.org. Accessed 8 Sept 2016.

  28. Department of Veterans Affairs Blueprint for Excellence, VHA Blueprint for Excellence. 2014. http://www.va.gov/HEALTH/docs/VHA_Blueprint_for_Excellence.pdf. Accessed 12 July 2016.

Download references

Acknowledgements

The views expressed are those of the authors and do not necessarily reflect those of the Department of Veterans Affairs, the United States Government, or the academic affiliate organizations. This work was supported by the VA IDEAS 2.0 HSRD Research Center and the CREATE: A VHA NLP Software Ecosystem for Collaborative Development and Integration #CRE 12–315.

Author information

Authors and Affiliations

  1. IDEAS Center SLC VA Healthcare System, 500 Foothill Drive, Salt Lake City, UT, 84148, USA

    Jennifer H. Garvin, Megha Kalsy, Guy Divita, Gregory Coronado, Doug Redd, Carrie Christensen, Brent Hill, Natalie Kelly & Qing Zeng Treitler

  2. GRECC SLC VA Healthcare System, 500 Foothill Drive, Salt Lake City, UT, 84148, USA

    Jennifer H. Garvin

  3. Division of Epidemiology, University of Utah School of Medicine, 295 Chipeta Way, Salt Lake City, UT, 84132, USA

    Jennifer H. Garvin

  4. Department of Biomedical Informatics, University of Utah School of Medicine, 421 Wakara Way, Ste. 140, Salt Lake City, UT, 84108, USA

    Jennifer H. Garvin, Megha Kalsy, Guy Divita, Doug Redd, Carrie Christensen, Brent Hill & Qing Zeng Treitler

  5. VA Connecticut Healthcare System, 950 Campbell Avenue, West Haven, CT, USA

    Cynthia Brandt

  6. Yale School of Medicine, 333 Cedar St., New Haven, CT, USA

    Cynthia Brandt

  7. James A Haley Veterans Hospital, 13000 Bruce B. Downs Blvd, Tampa, FL, USA

    Stephen L. Luther

  8. Department of Clinical Research and Leadership, George Washington University School of Medicine and Health Sciences, 2100 Pennsylvania Ave, NW, Washington, DC, 20037, USA

    Doug Redd & Qing Zeng Treitler

Authors
  1. Jennifer H. Garvin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Megha Kalsy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cynthia Brandt
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stephen L. Luther
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guy Divita
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Gregory Coronado
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Doug Redd
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Carrie Christensen
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Brent Hill
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Natalie Kelly
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Qing Zeng Treitler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jennifer H. Garvin.

Ethics declarations

Ethical statement

This study was approved by the VA Central IRB #CIRB 13–17 Zeng.

Funding

This work was supported by project number CRE 12–315 from the United States (U.S) Department of Veterans Affairs VA IDEAS 2.0 HSR&D Research Center. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs, their academic affiliates, or the U.S. Government.

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the University of Utah IRB and VA SLC Research and Development Office and with the 1964 Helsinski declaration and its later amendments or comparable ethical standards.

Informed consent

We obtained informed consent from all interview participants included in the study.

Additional information

This article is part of the Topical Collection on Systems-Level Quality Improvement

Appendix

Appendix

Table 1 Summary of functions and design recommendations
Full size table
Table 2 Summary of resources recommendations
Full size table
Table 3 Summary of information recommendations
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Garvin, J.H., Kalsy, M., Brandt, C. et al. An Evolving Ecosystem for Natural Language Processing in Department of Veterans Affairs. J Med Syst 41, 32 (2017). https://doi.org/10.1007/s10916-016-0681-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10916-016-0681-4

Keywords

Search

Navigation