Validating a decision support system for anti-epileptic drug treatment: Part I: initiating anti-epileptic drug treatment
Introduction
Physicians and nurses must trust the software they use, especially when it supports decision-making. It is therefore important that software is evaluated extensively. In the case of knowledge based systems a particular problem exists. Often an accepted golden standard to verify the knowledge base is not available. In such a case one usually takes the majority decision of a panel of experts as a reference.
This approach was also intended for the evaluation of a prototype system that supports the prescription of anti-epileptic drugs. However, in this domain the experts were less unanimous about the optimal treatment than was initially assumed by domain experts. A majority decision could not be reached for every case. A consensus meeting had to be organized to obtain a reference.
This contribution describes how the quality of a decision support system that is used in a domain where experts differ in their opinion about optimal decisions was assessed. This part of the paper focuses on the initiation of the drug treatment.
In the next section we provide some background about the domain. Then the system that supports the initiation of a treatment is presented. The development process of the paper cases, used by the experts and later on by the system for prescribing, is then described. After presenting the results of evaluating the system, the evaluation process is discussed in the last section.
Section snippets
Epilepsy
Epilepsy is a disorder of the brain that manifests itself through brief, usually unprovoked typical disturbances of behaviour, emotion, motor function or sensation. Once the diagnosis epilepsy is established a drug treatment has to be initiated. This entails the choice of an anti-epileptic drug, the determination of an adequate dose and the subsequent adjustment of the therapy [1], [2], [3], [4], [5], [6], [7], [8]. It is essential that the dose is high enough to suppress the seizures but low
Material and methods
The system was validated in a laboratory setting. Use was made of paper test cases. To evaluate the quality of a prescription provided by the system a reference is needed. Both the design of the test cases and the procedure to obtain a reference is described in this section.
Comparing the correctness of the group of neurologists and the system
As described above in 47 of the 60 cases a majority diagnosis (also called the reference diagnosis) was established. Only the prescriptions of the neurologists who agreed with this reference diagnosis were used in our analysis: 114 of the 141 prescriptions provided by the prescribing neurologists were based on the reference diagnosis. The neurologists prescribed five different drugs, whereas only three drugs were appropriate according to the consensus reference. The system only prescribed two
How good is the system as compared to the neurologists
To answer this question one has to consider two aspects: the selection of the drug and the daily dose. The neurologists in our study expected that the inter-individual variability with respect to the drug, given the diagnosis, would be relatively small. Also a pilot study among two neurologists not involved in the evaluation process showed a consistent use of different anti-epileptic drugs for different types of epilepsy. It was expected that there would be more variability in the daily dose.
Acknowledgements
We would like to thank the neurologists who participated in this study. Furthermore we thank the National Epilepsy Fund of The Netherlands and the Stichting Wetenschappelijk Onderzoek Limburg/Universiteitsfonds Limburg for their financial support.
References (14)
- et al.
A methodology for evaluation of knowledge-based systems in medicine
Artif. Intell. Med.
(1994) Standard approach to antiepileptic drug treatment in the United Kingdom
Epilepsia
(1994)Standard approach to antiepileptic drug treatment in the United States
Epilepsia
(1994)The drug treatment of epilepsy
Entering a new drug era: practical issues in the treatment of epilepsy
Neurology
(1995)- et al.
Antiepileptic drugs
New Engl. J. Med.
(1996) - Commission on Antiepileptic Drugs of the International League Against Epilepsy, Guidelines for therapeutic monitoring...
Cited by (8)
The Influence of Computer Resources on Child Neurology
2017, Swaiman's Pediatric Neurology: Principles and Practice: Sixth EditionDescription and feasibility of MINDSET: A clinic decision aid for epilepsy self-management
2013, Epilepsy and BehaviorCitation Excerpt :The emergence of electronic health (eHealth) applications to support patients in their daily self-management monitoring and decision-making is contributory to impacting epilepsy self-management [21]. In epilepsy, clinic-based decision support systems have primarily centered on diagnostic and pharmacologic support [22–29]. This is consistent with the general application of such systems in medicine that focus on the technical aspects of care, in contrast to the personal or social concerns of patients [30].
Development of a national protocol to screen Dutch cancer survivors on late cancer treatment effects
2007, International Journal of Medical InformaticsClinic-based mobile health decision support to enhance adult epilepsy self-management: An intervention mapping approach
2017, Frontiers in Public HealthEpilepsy - An ongoing challenge
2005, Acta Medica CroaticaInformation technology, the internet, and the future of neurology
2003, Neurologist