An experimental methodology to evaluate machine learning methods for fault diagnosis based on vibration signals

Highlights

• Systematic analysis of overoptimistic results in machine learning fault diagnosis.

• Computational framework to test new feature models.

• Computational framework to test new classifier architectures

• Experimental analysis of more realistic fault diagnosis scenarios.

Abstract

This paper presents a systematic procedure to fairly compare experimental performance scores for machine learning methods for fault diagnosis based on vibration signals. In the vast majority of related scientific publications, the estimated accuracy and similar performance criteria are the sole quality parameter presented. However, the experimental design giving rise to these results is mostly biased, based on unacceptably simple validation methods and on recycling identical patterns in test data sets, previously used for training. Moreover, the methods in general overfit their hyperparameters, introducing additional overoptimistic results. In order to remedy this defect, we critically analyse the usual training-validation-test division and propose an algorithmic guideline in the form of a validation framework. This allows a well defined comparison of experimental results. In order to illustrate the ideas of the paper, the Case Western Reserve University Bearing Data benchmark is used as a case study. Four distinct classifiers are experimentally compared, under gradually more difficult generalization tasks using the proposed evaluation framework: K-Nearest-Neighbor, Support Vector Machine, Random Forest and One-Dimensional Convolutional Neural Network. An extensive literature review suggests that most vibration based research papers, particularly for the Case Western Reserve University Bearing Data, use similar patterns for training and testing, making their classification an easy task.

Introduction

Software based fault diagnosis is an essential tool to guarantee the safety and maintainability of dynamic processes (Gao et al., 2015, Chiang et al., 2001). A principal distinction of employable methods is model-based fault diagnosis, c.f. for instance (Varga, 2017, Gertler, 2017, Ding, 2012), and model-free diagnosis, e.g. (Ding, 2016, McMillan and Vegas, 2019). Vibration based fault diagnosis focuses on analysing vibration signals in order to identify possible equipment faults. Some research works try to find signal characteristics or measures for identifying faults (Smith and Randall, 2015, Diaz et al., 2015, Diaz et al., 2017). Other research works focus on machine learning techniques that use the signals for training and testing classifiers for identifying faults. However, this work focuses on the latter approach.

The access to real world, well documented benchmark data is limited. A limited set of repositories with real vibration signals, obtained from existing mechanical systems, is available (Lee et al., 2007, Nectoux et al., 2012, Bechhoefer, 2016, Paderborn, 2020, MaFaulDa, 2016). The Case Western Reserve University (CWRU) Bearing Data (CWRU, 2014) is probably referred to the most in scientific literature. Some research papers use non publicly available vibration data sets (Liao et al., 2019, Lei et al., 2016, Verstraete et al., 2017).

An extensive amount of research works apply model-free, machine learning methods to classify distinct operational states of the process. A robust fault diagnosis system must be able to generalize well. This means that a trained classifier should be able to recognize as many faults as possible, even when there are variations of the machine conditions. Presented results must be reproducible and must be statistically significant; otherwise overoptimistic results are obtained. For instance, when a limited amount of samples are available, the data must be separated into non-overlapping sets. Each set may not be visible to the other set during the parametrization of the diagnosis system. It must be avoided that the same data is used both for tuning the hyperparameters of a classifier model and for testing the resultant classifier. Otherwise the results will probably be biased towards optimism. A typical example are the kernel and its associated variables in a Support Vector Machine (SVM). Which kernel is best for a C-SVM, RBF or Polynomial? And then, when using, for instance, the RBF kernel, use a grid search to find the best combination of the C regularization value and of the spread $\gamma$. Another example are deep learning structures of artificial neural networks where the layout is sometimes adjusted until it delivers the best results for the same data set. A more justified way of presenting performance scores is to isolate a test set completely. With the rest of the data, the hyperparameter tuning and the training can be done. When the final adjustments have been made, the test set is then used to estimate, for instance, an accuracy score. This procedure can be repeated several times, however the test set needs to always be kept apart. This hierarchy will be denoted as the inner and outer loop. Even then, when there is only a small amount of samples available for training, the performance may surpass theoretical Bayes limits.

Another common practice with vibration based machine learning research consists of defining a class, based on the chunks of a single chopped signal, even sometimes overlapping. Then, chunks of the same signal are used both on the training and testing data sets (Lei et al., 2016, Zhang et al., 2017, Verstraete et al., 2017, Liao et al., 2019). We call this the similarity bias problem. The patterns used for testing are almost indistinguishable from patterns used in the training data set. This fact may lead to an oversimplified model of real world fault diagnosis problems. A robust system must comprehend different machine conditions, and still be able to provide reasonable diagnostic information.

An important aspect when experimentally comparing fault diagnosis approaches consists of statistically verifying if the results are significantly different. However, this is seldom the case in the vibration based fault diagnosis research works.

The concern with the reproducibility of scientific research has steadily increased recently. Reproducibility is defined as obtaining consistent results using the same data and computational code as the original study. It has been found that many scientific studies are difficult or impossible to reproduce. The vibration based fault diagnosis works often have a lack of reproducibility.

This work proposes an experimental methodology for evaluating machine learning approaches for fault diagnosis based on vibration signals aiming to cope with all previously mentioned problems, i.e. completely isolating the test set, avoiding the similarity bias, verifying statistically significant differences and also allowing reproducibility. The main contributions of the paper are

• 1. A methodology for machine learning applied to vibration signal analysis, integrating nested cross validation, reproducibility, statistical analysis and avoiding similarity bias. Except for the latter, all these topics have already been applied in an isolated manner in the context to fault diagnosis;

• 2. An experimental study with synthetic data sets suggests superiority of the nested cross validation approach, especially for a small number of training samples;

• 3. The identification of common problems of research papers in the area of model-free fault diagnosis;

• 4. Identification of the overoptimistic bias, due to very similar training and test samples of the same class of machine conditions;

• 5. A study of the CWRU database, the principal real vibration signal source used in the scientific literature of the area.

The rest of the paper is organized in the following manner: Section 2 reviews conventional performance estimation methods in supervised learning problems. Furthermore, the common cross validation techniques are improved by a two-level evaluation hierarchy that isolates performance estimation from hyperparameter tuning. Synthetic data sets with known Bayesian error bounds are used to juxtapose the conventional and improved evaluation methods, suggesting the mostly overoptimistic results in research papers. Section 3 focuses on the important aspects of the proposed methodology for avoiding the similarity bias, for checking statistical significance of result differences, and also how to guarantee reproducibility. Section 4 critically reviews research works that use the vibration signals on machine learning approaches for fault diagnosis. Special focus is given to the CWRU mechanical data set, emphasizing the applied methods, and highlighting, if applicable, the defects that motivate the elaboration of this study. Moreover, the paper shows how the methodology may be customized for a specific data set. Section 5 and Section 6 present the case study of the CWRU Bearing Data Set. Section 5 defines different experimental modes for fault diagnosis, with gradually increasing difficulties. Section 6 provides an application of the proposed framework for the CWRU data, with four different classifier models, varying the generalization difficulty of the fault diagnosis, and finally the conclusions are drawn in Section 7.