Predicting the need for CT imaging in children with minor head injury using an ensemble of Naive Bayes classifiers
Introduction
Computed tomography (CT) is widely accepted as an effective diagnostic modality to detect rare but clinically significant intracranial injuries in patients suffering minor head injury. As such, it has been increasingly utilized as a routine test for these patients [1]. However, a seminal study by Brenner and Hall [2] warns against its harmful effects (particularly for children) due to the radiation exposure associated with CT. Independent CT imaging studies [1], [3], [4] advocate the adoption of a comprehensive approach that targets physicians’ education to reduce the over-reliance on CT imaging for head injury patients.
The diagnosis of a potentially serious brain injury following a minor head trauma is a well-documented challenge [5]. It is believed that clinical decision rules could help with this challenge and reduce unnecessary CT imaging. Broder [4] recommends that such decision rules rely on readily available patient data including physical examination and a patient's history. In line with these recommendations and in response to a growing need to improve the management of pediatric patients with minor head trauma in the emergency department (ED), Osmond et al. [6] developed the Canadian Assessment of Tomography for Childhood Head Injury (CATCH) clinical decision rule. The prospective cohort study was conducted in ten Canadian pediatric teaching hospitals and enrolled children brought to the ED who had blunt head trauma characterized by loss of consciousness, amnesia, disorientation or repeated vomiting along with a score of at least 13 on the Glasgow Coma Scale. Such patients have often, but not always in a consistent manner [7], been referred to CT imaging to rule out a potential intracranial lesion that might necessitate a neurologic intervention.
The CATCH data set contains 3866 patient records described by 26 clinical attributes (standardized clinical findings from a patient's medical history, general examination and neurological status). These patient records are partitioned according to two classification schemes; the primary classification distinguishes between patients who had a brain injury and those who had no injury, where “brain injury” is defined as any acute intracranial finding revealed on a CT image and attributable to acute head trauma. Because this classification corresponds directly to the need for CT imaging (patients with the suspected injury require this test, and the remaining ones do not), we label these two classes as CT = yes and CT = no respectively. The secondary classification indicates whether or not a neurologic intervention was needed, and hence, we refer to these two secondary classes as neurologic intervention = yes and neurologic intervention = no. It is important to note that records in the neurologic intervention = yes class form a subset of the CT = yes class. Retrospectively, the need for neurological intervention was defined in the CATCH data set by the death of the patient within a week after the head injury or by the need of any of the following procedures within the same time period: craniotomy, an elevation of skull fracture, intracranial pressure monitoring, or intubation for head injury (demonstrated on the CT image).
In order to assess the physician's perception on the use of a clinical decision rule for minor head trauma patients, Osmond conducted a survey among Canadian pediatric ED physicians to determine a clinically acceptable level of prediction performance, so that, ED physicians will be confident with the rule. Results of this survey (personal communication, 80% response rate) revealed that the detection of a serious intracranial lesion is important for clinicians. Consequently, the CATCH study targeted to achieve a sensitivity of 95% when predicting the need for CT imaging. For those patients who subsequently required neurologic intervention, the CATCH rule aimed for 98% sensitivity.
With these findings in mind, Osmond and colleagues set to create a decision rule that maximized sensitivity of prediction at an inherent cost to specificity. Using recursive partitioning they developed a rule that is clear and intuitive for ED physicians to apply for the identification of two levels of risk among children with minor head trauma. According to the derived rule, the CT decision is made through a stratified evaluation of a patient's risk factors, where the presence of any of these factors indicates the need for CT imaging to detect a serious injury. The structure of the CATCH rule is presented in Fig. 1. This rule can be interpreted as a disjunction of the risk factors as the rule's premise, and the decision to perform CT imaging as a conclusion. In case of the high risk factors, the conclusion also indicates that neurosurgical intervention is necessary. Osmond et al. evaluated the performance of the CATCH rule on 1000 bootstrapped tests and reported the sensitivity and the specificity of the high risk (top four factors in the CATCH rule) for neurologic intervention as 97.9% and 70.2% respectively. They also reported the sensitivity and the specificity of all risk factors for the need of CT imaging to detect any brain injury as 98.1% and 50.0% respectively.
We were granted a unique opportunity to work with CATCH data to develop a prediction model that indicates the need for CT imaging. Following the CATCH study, Osmond and colleagues have initiated a prospective evaluation of the CATCH rule in selected Canadian hospitals. For this evaluation, patient information was limited to 17 out of the original 26 attributes. In order to maintain compatibility and continuity of the CATCH study, we decided to use the same 17 attributes for the construction of our prediction models from the CATCH data. In this way, the model discussed in the paper can be tested again when prospectively collected data becomes available.
The decision of whether a minor head injury patient requires CT imaging is a binary classification problem. The objective is to distinguish between patients who require a CT scan (CT = yes) and those who do not (CT = no). Thus, our research question is: can a balanced (in terms of sensitivity and specificity) and well performing prediction model be automatically derived from the CATCH data? As a corollary to this question, we do not constrain the prediction model with respect to its interpretability and comprehensibility by non-computer science experts.
An argument for having a balanced prediction model relies on the need to mitigate long-term effects of ionizing radiation associated with the potential overuse of CT imaging that might occur when maximization of sensitivity drives model's development. We are aware that the CATCH rule developed in conjunction with physicians’ expertise according to a conservative approach is likely to outperform (in terms of sensitivity) an automatically constructed prediction model. However, we believe that such model may help in establishing reference performance indicators for the CATCH rule and estimating a trade-off between the sensitivity and specificity of prediction.
Additionally, we want to show how to automatically develop a prediction model from severely imbalanced data. This class imbalance situation is commonly encountered when analyzing clinical data where the population of patients with an acute health condition is usually significantly smaller than the population of relatively healthy ones. Our research demonstrates that well-performing model can be developed by utilizing data under-sampling when constructing an ensemble prediction classifier composed of multiple Naive Bayes (NB) classifiers.
While the CATCH study explicitly identifies a high-risk subgroup of those patients who need neurologic intervention (neurologic intervention = yes class), we do not make this distinction, and therefore, we do not consider maximal sensitivity of prediction for this group to be a driving objective for the development of our model. However, for the purpose of consistency with Osmond's study, we report separately, the model's performance for patients in the neurologic intervention = yes class (i.e., the high risk patients according to the CATCH rule).
The paper is organized as follows. In the next section, we present related research on applying data mining techniques to clinical problems. Section 3 describes the data set used in this research, briefly characterizes data mining methods selected for the study, and reviews the experimental design. Section 4 presents experimental results, and the last section concludes with a discussion.
Section snippets
Related research
Data mining techniques allow for the development of sophisticated prediction models capable of analyzing high-dimensional data [8] without relying on domain expertise during the model development process. Techniques that are suitable for medical domains are discussed and summarized in [9] – they include rule and decision tree induction, instance-based learning, Bayesian classification, and inductive logic programming.
Clinical data that describes a specific patient condition or disease poses
Data set
Attributes describing the CATCH data, which were used in our analysis, are listed in Table 1. We applied an automatic approach to discretizing values for Age and with the aid of a clinical expert we discretized values for VomitNum. Both discretizations were verified and approved by physicians involved in the CATCH study. To replicate the CATCH study design, we imputed missing attribute values with clinically reasonable values (they usually corresponded to a negative answer, e.g., no or none in
Evaluation of the E-NB model
Table 3 contains evaluations of E-NB and the three other ensemble models. It shows the mean and standard deviations of the evaluation measures as well as their confidence intervals (CI) with 95% confidence.
The E-NB model outperformed other models in terms of AUC values, thus it demonstrated the best capability to separate decision classes, and all differences between AUC values were statistically significant. E-NB was also best in terms of G-mean, only the E-TB model achieved comparable value –
Discussion
The decision to order a diagnostic test and the timing of this test are two important facets of medical decision-making. The CATCH rule was developed to help identify children with minor head injury who require CT imaging. It was created from prospectively collected data and designed to eliminate the false negatives for critical patients who require neurologic intervention. Therefore, the rule's performance is characterized by an almost perfect sensitivity at a cost of low specificity.
Having
Conflict of interest statement
No conflicts of interest exist.
Acknowledgements
The authors would like to thank Terry P. Klassen MD, George A. Wells PhD, Rhonda Correll RN, Anna Jarvis MD, Gary Joubert MD, Benoit Bailey MD, Laurel Chauvin-Kimoff MD CM, Martin Pusic MD, Don McConnell MD, Cheri Nijssen-Jordan MD, Norm Silver MD, Brett Taylor MD, Ian G. Stiell MD; of the Pediatric Emergency Research Canada (PERC) Head Injury Study Group for providing access to the CATCH data.
The current version of the paper benefited from the insightful comments of the reviewers.
This research
References (45)
Selected techniques for data mining in medicine
Artif Intell Med
(1999)- et al.
Uniqueness of medical data mining
Artif Intell Med
(2002) Fast effective rule induction
- et al.
Minor head injury: CT-based strategies for management – a cost-effectiveness analysis
Radiology
(2010) - et al.
Computed tomography – an increasing source of radiation exposure
N Engl J Med
(2007) - et al.
Reducing inappropriate diagnostic practice through education and decision support
Int J Qual Health Care
(2010) CT utilization: the emergency department perspective
Pediatr Radiol
(2008)- et al.
Poor prediction of positive computed tomographic scans by clinical criteria in symptomatic pediatric head trauma
Pediatrics
(1987) - et al.
CATCH: a clinical decision rule for the use of computed tomography in children with minor head injury
CMAJ
(2010) - et al.
Variation in utilization of computed tomography scanning for the investigation of minor head trauma in childr a Canadian experience
Acad Emerg Med
(2000)
Machine learning for detection and diagnosis of disease
Annu Rev Biomed Eng
Severe class imbalance: why better algorithms aren’t the answer
Learning from imbalanced data
IEEE Trans Knowl Data Eng
Data mining for imbalanced data sets: an overview
C4.5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling
SMOTE: synthetic minority over-sampling technique
J Artif Intell Res
Addressing the curse of imbalanced training sets: one-sided selection
Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning
The class imbalance problem in pattern classification and learning
The boosting approach to machine learning: an overview
Exploiting the cost (in)sensitivity of decision tree splitting criteria
AdaCost: misclassification cost-sensitive boosting
Cited by (23)
Deep Neural Networks Predict the Need for CT in Pediatric Mild Traumatic Brain Injury: A Corroboration of the PECARN Rule
2022, Journal of the American College of RadiologyCitation Excerpt :The DANN model was trained with the 15 predictors used to drive the PECARN-A and PECARN-B rules for children younger than 18 years. The performance was superior to that of state-of-the-art models that replicated such clinical rules in terms of balanced sensitivity of 98.6% and specificity of 99.7% [26,27]. Table 10 lists the limited studies that have applied ML to mimic the developed mTBI clinical decision rules and predict the need for CT scans in different populations.
The detection of mild traumatic brain injury in paediatrics using artificial neural networks
2021, Computers in Biology and MedicineKnowledge discovery in medicine: Current issue and future trend
2014, Expert Systems with ApplicationsCitation Excerpt :This was done based on decision tree, multilayer perceptron and general regression neural network. Naïve bayes can be used for predict the need for CT scanning in children with minor head injuries (Klement et al., 2012). Crash scene data can be used to evaluate severe and vehicular injuries by CART algorithm.
Influence of data discretization on efficiency of Bayesian classifier for authorship attribution
2014, Procedia Computer ScienceKnowledge Discovery in Biomedical Data: Theory and Methods
2013, Methods in Biomedical Informatics: A Pragmatic ApproachMild head injury in pediatrics: Algorithms for management in the ED and in young athletes
2013, American Journal of Emergency MedicineCitation Excerpt :However, the rate of cranial CT is still around 30% [2] and maybe even higher in certain populations or practice settings [48]. Indiscriminate use of cranial CT should not be substituted for an experienced physician's clinical judgment [49]. The rate of sport-related minor head injuries or concussions is underestimated because they often go unreported and are not recognized by young athletes, and often only concussions with loss of consciousness are regarded as significant [8].