Extracting decision trees from trained neural networks
Abstract
In this paper we present a methodology for extracting decision trees from input data generated from trained neural networks instead of doing it directly from the data. A genetic algorithm is used to query the trained network and extract prototypes. A prototype selection mechanism is then used to select a subset of the prototypes. Finally, a standard induction method like ID3 or C5.0 is used to extract the decision tree. The extracted decision trees can be used to understand the working of the neural network besides performing classification. This method is able to extract different decision trees of high accuracy and comprehensibility from the trained neural network.
References (0)
Cited by (101)
A multivariate Markov chain model for interpretable dense action anticipation
2024, NeurocomputingDense action anticipation, as opposed to next action anticipation, deals with predicting multiple actions over a long horizon of a few minutes. Recent approaches for dense action anticipation are based on deep learning, which lacks interpretability in that the models cannot explain the decisions made through a causal relationship between past observations and predictions. In this paper, we propose a Goal oriented multivariate Markov chain (GoMMC) for interpretable dense action anticipation, which can capture the influence between various objects and their interactions, allowing for a probabilistic selection of actions performed in the long term. Experiments on 50Salads and Breakfast datasets show that the proposed model performs better than deep learning models when ground truth information on past objects and actions is available.
Explainable cyber threat behavior identification based on self-adversarial topic generation
2023, Computers and SecurityCyber Threat Intelligence (CTI) provides ample evidence and information regarding the detection of cyber attack activities. Existing methods employ CTI reports to extract Tactics, Techniques and Procedures (TTPs) for attack detection. Nevertheless, these methods are challenged in providing necessary and sufficient evidentiary support for recognition decisions, making it difficult for human operators to comprehend and accept the decision-making process. This paper proposes a topic prototype-based explainable TTPs classification approach, which provides accurate boundaries for key evidences to justify the results of TTPs classification. The proposed method introduces a self-adversarial framework for obtaining necessary and sufficient evidence for TTPs classification. The framework consists of an evidence generator and a TTPs classifier discriminator. The evidence generator utilizes a topic prototype-based keyword importance filtering method to extract evidence from CTI text while removing noise, resulting in an evidence set and a perturbation set. Subsequently, the impact of the evidence set and the perturbation set on TTPs classification is assessed using our siamese discriminator. The discriminator is specifically trained to ensure that only the elements belonging to the evidence set are accurately classified as TTPs information. The experiments primarily test the necessity and sufficiency of TTPs and evidence. In the sufficiency evaluations, classical deep learning methods are used for TTPs classification to verify the accuracy of the results, where the proposed method improves the Micro F1 scores by 0.16% to 6.63% and Macro F1s by 0.26% to 6.85%. To prove necessity, various case-based explainable methods are used to measure the completeness of CTI evidence. The results shows that the proposed method is able to obtain more stable prediction, more reasonable evidence sets, and more significant boundaries.
Interpretability in the medical field: A systematic mapping and review study
2022, Applied Soft ComputingRecently, the machine learning (ML) field has been rapidly growing, mainly owing to the availability of historical datasets and advanced computational power. This growth is still facing a set of challenges, such as the interpretability of ML models. In particular, in the medical field, interpretability is a real bottleneck to the use of ML by physicians. Therefore, numerous interpretability techniques have been proposed and evaluated to help ML gain the trust of its users.
This review was carried out according to the well-known systematic map and review process to analyze the literature on interpretability techniques when applied in the medical field with regard to different aspects: publication venues and publication year, contribution and empirical types, medical and ML disciplines and objectives, ML black-box techniques interpreted, interpretability techniques investigated, their performance and the best performing techniques, and lastly, the datasets used when evaluating interpretability techniques.
A total of 179 articles (1994–2020) were selected from six digital libraries: ScienceDirect, IEEE Xplore, ACM Digital Library, SpringerLink, Wiley, and Google Scholar. The results showed that the number of studies dealing with interpretability increased over the years with a dominance of solution proposals and experiment-based empirical type. Diagnosis, oncology, and classification were the most frequent medical task, discipline, and ML objective studied, respectively. Artificial neural networks were the most widely used ML black-box techniques investigated for interpretability. Additionally, global interpretability techniques focusing on a specific black-box model, such as rules, were the dominant explanation types, and most of the metrics used to evaluate interpretability were accuracy, fidelity, and number of rules. Moreover, the variety of the techniques used by the selected papers did not allow categorization at the technique level, and the high number of the sum of evaluations (671) of the articles raised a suspicion of subjectivity. Datasets that contained numerical and categorical attributes were the most frequently used in the selected studies.
Further effort is needed in disciplines other than diagnosis and classification. Global techniques such as rules are the most used because of their comprehensibility to doctors, but new local techniques should be explored more in the medical field to gain more insights into the model’s behavior. More experiments and comparisons against existing techniques are encouraged to determine the best performing techniques. Lastly, quantitative evaluation of interpretability and physicians’ implications in interpretability techniques evaluation is highly recommended to evaluate how the techniques will perform in real-world scenarios. It can ensure the soundness of the techniques and help gain trust in black-box models in medical environments.
A large-scale comparison of Artificial Intelligence and Data Mining (AI&DM) techniques in simulating reservoir releases over the Upper Colorado Region
2021, Journal of HydrologyIn recent years, the Artificial Intelligence and Data Mining (AI&DM) models have become popular tools in assisting various aspects of reservoir operation. However, the practical uses are still rarely reported. Comparison experiment of many AI&DM models over a large number of reservoir cases is particularly valuable to help reservoir operators first examine the usefulness and transferability of different AI&DM models, and then identify the most stable and reliable AI&DM model in assist of various decision-making processes. In this study, a total of 12 AI&DM models with different parameterizations and simulation scenarios are comprehensively tested out and compared in simulating the controlled reservoir outflows of 33 reservoir cases over the Upper Colorado Region, United States. Results show that the Random Forecast and the Long-Short-Term-Memory model could consistently derive the best statistical performance than other models under the baseline simulation scenario. The employed AI&DM models could obtain satisfactory statistical interquartile ranges (25–75%) between [0.6–0.9], [0.3–0.8], and [0.2–0.8], for CORR, NSE, and KGE measurements, respectively, and [1.5–6.5], [−15 to 20], and [0.5–8.5] for the normalized RMSE, PBIAS and RSR measurements, respectively. Results also show Multi-Layer Perceptron model and Extreme Gradient Boosting Tree Algorithm produced more stable and superior performance than other models under more complex input scenarios. We also found that the performance of different AI&DM models are closely relevant to the reservoir elevations, sizes, and functionalities. Discussions were made about the sensitivity of AI&DM models’ parameterizations and the key advantages of AI&DM models over the rule-based reservoir models. We further identify that the main advantage of AI&DM models is the flexibility in designing input structures, whereas the rule-based simulation model is rather limited. Future studies were suggested regarding the best way reservoir operators and researchers could use, select, and apply different AI&DM models in simulating reservoir releases under different natural and modeling environments. This comparison study also serves as a reference and a piece of groundwork for further promoting the practical uses of AI&DM models in assisting reservoir operation.
Explaining neural networks without access to training data
2024, Machine Learning
- 1
Presently with CAIR, Raj Bhavan Circle, High Grounds, Bangalore 560 001, India