Simulation and sensor data fusion for machine learning application

Abstract

The performance of machine learning algorithms depends to a large extent on the amount and the quality of data available for training. Simulations are most often used as test-beds for assessing the performance of trained models on simulated environment before deployment in real-world. They can also be used for data annotation, i.e, assigning labels to observed data, providing thus background knowledge for domain experts. We want to integrate this knowledge into the machine learning process and, at the same time, use the simulation as an additional data source. Therefore, we present a framework that allows for the combination of real-world observations and simulation data at two levels, namely the data or the model level. At the data level, observations and simulation data are integrated to form an enriched data set for learning. At the model level, the models learned from observed and simulated data separately are combined using an ensemble technique. Based on the trade-off between model bias and variance, an automatic selection of the appropriate fusion level is proposed. Our framework is validated using two case studies of very different types. The first is an industry 4.0 use case consisting of monitoring a milling process in real-time. The second is an application in astroparticle physics for background suppression.

Introduction

The performance of machine learning depends to a great extent on the quality and the quantity of data available for training [1]. Since large data sets are most often required for training, the fusion of data sets from many sources can be helpful, but also challenging [2]. The problems of duplicate detection [3], schema matching [4], and conflict resolution [3], [5] are to be solved for the integration of several databases, especially when enclosing heterogeneous types of data. Integrating data from many sensors in order to get a comprehensive data set is widely studied for multi-sensor systems fusion, see, e.g. [6], [7]. With the internet of things, a multitude of distributed data sets are available for learning [8]. There, the different sensors measure different features of the same event, so that a new combined feature set can be created for training a model [8].

More recently, generative neural network models have been used to construct synthetic data sets [9]. However, in this paper, we want to focus on synthetic data from simulations. Simulations are most often considered as the ground truth, because they are based on theoretical knowledge of the particular domain. Hence, they are used for testing learned models or annotating the data. Simulations are used as experiments that offer scientists the chance to study a range of phenomena in a structured way. This is a standard procedure in computer science (e.g., [10]) as well as in engineering [11] and physics [12], to name but a few. Many investigations of multi-sensor fusion have used simulation environments only for testing their frameworks [13], [14]. Process simulations are established instruments to investigate processes in a virtual environment prior to deployment [15], [16]. More recently, simulations have been conceived as powerful data generators, providing promising opportunities for simulation data mining [17], [18]. However, simulations have many limitations in practice. They cannot provide a completely accurate representation of reality in real-time [19] and have usually only a limited prediction accuracy when modeling complex relationships [20].

In contrast, machine learning models can be applied in real-time and offer the opportunity to predict events based on the analysis of a set of explanatory variables. Therefore, new trends aim at replacing simulation models with surrogate machine learning models that have been trained on simulation data [21]. Some recent works focuses on learning from simulation data to monitor the real-world process and predict upcoming unknown events with a reasonable accuracy [18], [22], [16]. However, none of these approaches used simulation data to enrich the sensor data for a given learning task. In this paper, we present a framework that formally validate the use of simulation as a synthetic data generator by fulfilling certain conditions, namely completeness, conciseness, and correctness. It also suggests to solve possible data mismatches between sensors and simulation using machine learning-based methods. Additionally, our framework challenges the typical data and model handling for machine learning in the sense that it allows for more flexibility in the combination of sensor data and simulation. In particular, it automatically selects between two different integration levels.

Data-level fusion

The integration of observations and simulation is not restricted to the raw data of observations. Instead, a common representation for both is created and then enhanced by feature extraction, generation and selection. The performance of the model which is trained on these enhanced data is the criterion that guides the feature engineering.

Model-level fusion

The simulation and the sensor data can be used independently for training single models, one for each data set. The resulting models may then be combined to output the prediction. Weighting the two models adapts the framework to the application at hand.

Level choice decision

Moreover, the framework automatically decides which level to take based on a derived threshold for the co-variance of the single models. The criterion states that the variance-based error will be reduced if transiting from the data to the model level, if the co-variance is lower than the threshold.

The paper first gives an overview of related work in data fusion in Section 2. Conditions for the use of simulation as a valid data source are detailed in Section 3. The novel framework indicating the steps and methods of simulation-sensor fusion, including data mismatch evaluation and solving and integration level selection, is presented in Section 4. The real-world case study is then described as instance of the framework in Section 5. The real-world case studies are then described as instances of the framework in Section 5. The learning performance with and without the integration of simulation and sensor data is shown in the experiments and the automatic fusion level choice is evaluated.