Skip to main content
Springer Nature Link
Log in

Challenges in Evaluating Complex Decision Support Systems: Lessons from Design-a-Trial

  • Conference paper
  • First Online:
Artificial Intelligence in Medicine (AIME 2001)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 2101))

Included in the following conference series:

  • 1141 Accesses

Abstract

Decision support system developers and users agree on the need for rigorous evaluations of performance and impact. Evaluating simple reminder systems is relatively easy because there is usually a gold standard of decision quality. However, when a system generates complex output (such as a critique or graphical report), it is much less obvious how to evaluate it. We discuss some generic problems and how one might resolve them, using as a case study Design- a-Trial, a DSS to help clinicians write a lengthy trial protocol.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Wyatt, J., Altman, D., Heathfield, H., Pantin, C.: Development of Design-a-Trial, a Knowledge-based Critiquing System for Authors of Clinical Trial Protocols. Comp. Meth. Prog. Biomed. 43 (1994) 283–291

    Article  Google Scholar 

  2. Modgil, S., Hammond, P., Wyatt, J.C., Potts, H.: The Design-a-Trial Project: Developing a Knowledge-based Tool for Authoring Clinical Trial Protocols. In: 1st European Workshop on Computer-based Support for Clinical Guidelines and Protocols (EWGLP-2000). IOS Press, Amsterdam (in press)

    Google Scholar 

  3. Fox, J., Johns, N., Rahmanzadeh, A., Thompson, R.: ProForma: A General Technology for Clinical Decision Support Systems. Comp. Meth. Prog. Biomed. 54 (1997) 59–67

    Article  Google Scholar 

  4. Friedman, C., Wyatt, J.: Evaluation Methods in Medical Informatics. Springer Verlag, New York (1997)

    Google Scholar 

  5. Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., Walsh, S.: Assessing the Quality of Randomized Controlled Trials. Control. Clin. Trials 16 (1995) 62–73

    Article  Google Scholar 

  6. Bose, D. K., Heathfield, H. A., Andrew, M.: Collective Decision Problems in Medicine — A Basic Approach Looking for Cross-Fertilization in Clinical Surgery. Theoretical Surg. 7 (1992) 186–193

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Public Policy, University College London, 29/30 Tavistock Square, London, WC1H 9QU, UK

    Henry W. W. Potts & Jeremy C. Wyatt

  2. Centre for Statistics in Medicine, Institute of Health Sciences, Old Road, Oxford, OX3 7LF, UK

    Douglas G. Altman

Authors
  1. Henry W. W. Potts
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeremy C. Wyatt
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Douglas G. Altman
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science, University of Pavia, Via Ferrata, 1, 27100, Pavia, Italy

    Silvana Quaglini

  2. Departamento de Informática Faculdade de Ciências e Tecnologia, Universidade Nova de Lisboa, 2825-114, Caparica, Portugal

    Pedro Barahona

  3. Department of Medical Informatics and Image Analysis, Aalborg University, Fredrik Bajersv ej 7D, 9220, AlborgØst, Denmark

    Steen Andreassen

Rights and permissions

Reprints and permissions

Copyright information

© 2001 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Potts, H.W.W., Wyatt, J.C., Altman, D.G. (2001). Challenges in Evaluating Complex Decision Support Systems: Lessons from Design-a-Trial. In: Quaglini, S., Barahona, P., Andreassen, S. (eds) Artificial Intelligence in Medicine. AIME 2001. Lecture Notes in Computer Science(), vol 2101. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-48229-6_61

Download citation

  • DOI: https://doi.org/10.1007/3-540-48229-6_61

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-42294-5

  • Online ISBN: 978-3-540-48229-1

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics