Simpler is better: Lifting interpretability-performance trade-off via automated feature engineering

Highlights

• Complex models are not always better.

• SAFE extracts interpretable features from complex models.

• Complex models can be automatically simplified without loss of performance.

Abstract

Machine learning has proved to generate useful predictive models that can and should support decision makers in many areas. The availability of tools for AutoML makes it possible to quickly create an effective but complex predictive model. However, the complexity of such models is often a major obstacle in applications, especially in terms of high-stake decisions. We are experiencing a growing number of examples where the use of black boxes leads to decisions that are harmful, unfair or simply wrong. In this paper, we show that very often we can simplify complex models without compromising their performance; however, with the benefit of much needed transparency.

We propose a framework that uses elastic black boxes as supervisor models to create simpler, less opaque, yet still accurate and interpretable glass box models. The new models were created using newly engineered features extracted with the help of a supervisor model. We supply the analysis using a large-scale benchmark on several tabular data sets from the OpenML database. There are tree main results of this paper: 1) we show that extracting information from complex models may improve the performance of simpler models, 2) we question a common myth that complex predictive models outperform simpler predictive models, 3) we present a real-life application of the proposed method.