Validating a decision support system for anti-epileptic drug treatment: Part I: initiating anti-epileptic drug treatment

https://doi.org/10.1016/S1386-5056(99)00051-9Get rights and content

Abstract

In this contribution the validation of a prototype decision support system that implements a model of expertise for initiating anti-epileptic drug treatment is described. Since domain experts were of the opinion that prescribing was a rather straightforward process we used only one expert neurologist for knowledge elicitation. To determine the correctness of the system we intended to compare the contents of the system’s prescriptions with the majority decision of three neurologists. Because of a large variation in prescribing a majority decision could not be obtained in many cases. Even a Delphi procedure did not yield a majority decision in a large number of cases. Therefore a consensus meeting was organised to discuss cases where discrepancies remained. In the process the participating neurologists formulated prescription guidelines. These guidelines were used as a reference to determine the correctness of the prescriptions of both the system and of the neurologists. The acceptability of all prescriptions for each case was rated by the two neurologists who did not write a prescription for that case. From both comparisons it could be concluded that the system was at least as good in prescribing as individual neurologists.

Introduction

Physicians and nurses must trust the software they use, especially when it supports decision-making. It is therefore important that software is evaluated extensively. In the case of knowledge based systems a particular problem exists. Often an accepted golden standard to verify the knowledge base is not available. In such a case one usually takes the majority decision of a panel of experts as a reference.

This approach was also intended for the evaluation of a prototype system that supports the prescription of anti-epileptic drugs. However, in this domain the experts were less unanimous about the optimal treatment than was initially assumed by domain experts. A majority decision could not be reached for every case. A consensus meeting had to be organized to obtain a reference.

This contribution describes how the quality of a decision support system that is used in a domain where experts differ in their opinion about optimal decisions was assessed. This part of the paper focuses on the initiation of the drug treatment.

In the next section we provide some background about the domain. Then the system that supports the initiation of a treatment is presented. The development process of the paper cases, used by the experts and later on by the system for prescribing, is then described. After presenting the results of evaluating the system, the evaluation process is discussed in the last section.

Section snippets

Epilepsy

Epilepsy is a disorder of the brain that manifests itself through brief, usually unprovoked typical disturbances of behaviour, emotion, motor function or sensation. Once the diagnosis epilepsy is established a drug treatment has to be initiated. This entails the choice of an anti-epileptic drug, the determination of an adequate dose and the subsequent adjustment of the therapy [1], [2], [3], [4], [5], [6], [7], [8]. It is essential that the dose is high enough to suppress the seizures but low

Material and methods

The system was validated in a laboratory setting. Use was made of paper test cases. To evaluate the quality of a prescription provided by the system a reference is needed. Both the design of the test cases and the procedure to obtain a reference is described in this section.

Comparing the correctness of the group of neurologists and the system

As described above in 47 of the 60 cases a majority diagnosis (also called the reference diagnosis) was established. Only the prescriptions of the neurologists who agreed with this reference diagnosis were used in our analysis: 114 of the 141 prescriptions provided by the prescribing neurologists were based on the reference diagnosis. The neurologists prescribed five different drugs, whereas only three drugs were appropriate according to the consensus reference. The system only prescribed two

How good is the system as compared to the neurologists

To answer this question one has to consider two aspects: the selection of the drug and the daily dose. The neurologists in our study expected that the inter-individual variability with respect to the drug, given the diagnosis, would be relatively small. Also a pilot study among two neurologists not involved in the evaluation process showed a consistent use of different anti-epileptic drugs for different types of epilepsy. It was expected that there would be more variability in the daily dose.

Acknowledgements

We would like to thank the neurologists who participated in this study. Furthermore we thank the National Epilepsy Fund of The Netherlands and the Stichting Wetenschappelijk Onderzoek Limburg/Universiteitsfonds Limburg for their financial support.

References (14)

  • K. Clarke et al.

    A methodology for evaluation of knowledge-based systems in medicine

    Artif. Intell. Med.

    (1994)
  • D. Chadwick

    Standard approach to antiepileptic drug treatment in the United Kingdom

    Epilepsia

    (1994)
  • J.M. Pellock

    Standard approach to antiepileptic drug treatment in the United States

    Epilepsia

    (1994)
  • S. Shorvon

    The drug treatment of epilepsy

  • F.H. Allen

    Entering a new drug era: practical issues in the treatment of epilepsy

    Neurology

    (1995)
  • M.J. Brodie et al.

    Antiepileptic drugs

    New Engl. J. Med.

    (1996)
  • Commission on Antiepileptic Drugs of the International League Against Epilepsy, Guidelines for therapeutic monitoring...
There are more references available in the full text version of this article.

Cited by (8)

  • The Influence of Computer Resources on Child Neurology

    2017, Swaiman's Pediatric Neurology: Principles and Practice: Sixth Edition
  • Description and feasibility of MINDSET: A clinic decision aid for epilepsy self-management

    2013, Epilepsy and Behavior
    Citation Excerpt :

    The emergence of electronic health (eHealth) applications to support patients in their daily self-management monitoring and decision-making is contributory to impacting epilepsy self-management [21]. In epilepsy, clinic-based decision support systems have primarily centered on diagnostic and pharmacologic support [22–29]. This is consistent with the general application of such systems in medicine that focus on the technical aspects of care, in contrast to the personal or social concerns of patients [30].

  • Epilepsy - An ongoing challenge

    2005, Acta Medica Croatica
View all citing articles on Scopus
View full text