Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Nowadays, technical systems consist of components from various domains: mechanics, electronics, control engineering, and information technology. Their interaction is expressed by the term mechatronics. The basis of their action is established by collecting information about the system and its environment by sensors, processing and analyzing this information in order to generate a suitable reaction, and adapting the behavior of the overall system. If this behavior is implemented in such a way that no human control is required for system operation, all behavioral intelligence and decisions are performed by the system itself. We call such systems autonomous system. Research in the domain of autonomous systems has gained great attention in the recent years.

Obviously the development process of such autonomous systems is different [4] as the key challenge is the implementation of the decision making part. Because these systems show up their powerfulness primary at their operation, the evaluation of the system in operation becomes essential to enable analysis, validation and performance assessment. In critical (including safety-critical as well as mission-critical) application domains, for example, execution of the system for testing or evaluation purposes may involve high risk with potentials to damage the system itself or its environment or even endanger human life. Additionally, operating the real life system for evaluation purposes might also be expensive, also having in mind potential damages or dangers.

Virtual reality (VR) technology offers an opportunity to manage the execution of such systems and provides visualization techniques that are used to present the system behavior respectively. VR puts the developer responsible for evaluation into a virtual three-dimensional world and enables to use 1:1 virtual 3D model of the product shape and its real environment. The kinematics behavior of the system is visualized using animations or/and further visualization techniques, which may also include the visualization of invisible physical behavior. With all its characteristics, VR provides an adequate platform for the evaluation of a system in operating mode.

2 Evaluation of Autonomous Approaches

Autonomous Systems are computational systems that differ from conventional systems mainly by their decision making process: the decisions made by the system are not human-controlled but are performed by the system itself in an autonomous manner. The autonomous decisions process relies on data obtained by sensing of the system itself and its environment, and is based on predefined rules that may be regulated by random factors also, in order to meet the system’s specified objective function. This makes the evaluation of such approaches challenging. The objective of this paper is to examine the evaluation of such novel autonomous approaches by virtual environments.

2.1 Evaluation Target

We have developed an Online Anomaly Detection for self-reconfigurable Real-time operating systems (RTOS) that operates fully autonomously and is able to cope with dynamically changing behavior caused by self-reconfigurability (which is a property of autonomous systems). The approach of our Online Anomaly Detection was inspired by Danger Theory from Artificial Immune Systems [1] that have been derived from the behavior of the Human Immune System. Danger Theory features adaptability by its nature and can be used for pattern matching, classification, anomaly detection, etc. It introduces so called health signals that allow to autonomously classify novel, dynamically generated and previously unknown behaviors correctly, without relying on a clearly separated training phase which usually is required in anomaly detection.

The health signals make it possible to implement an autonomous context-related anomaly detection: The health signals reflect the operating system state based on an integrated monitoring of the system components and their performance in operation. The current task’s and system behavior is evaluated by means of the obtained health signals: A behavior pattern is declared as normal behavior if the input signals indicate a healthy state of the OS, while, in case of danger signals, the evaluation of behavioral patters lead to an anomaly alert. By this approach, also novel and previously unknown behaviors can be classified in a reliable manner, as in contrary to classic anomaly detection, this context-related approach prevents to categorically classify unknown behavioral patterns as unsafe or malicious anomalies. Detailed concepts of the approach have been published at IEEE workshops [10] and [9] on self-organizing real-time systems. These concepts require exhaustive evaluation now.

2.2 Evaluation Methodologies

Evaluation of autonomous approaches in not straightforward because of the role of decision making process [8]. Therefore, test methodologies of ordinary systems are not suitable, but no standardized or state-of-the-art method exist up to now. Several publications by Roske et al. [8], Thompson [12], and Garrett [3] analyze the problems and challenges concerned with evaluation and testing of autonomous systems. Pure static evaluation by mathematical methods or formal verification is not sufficient as the strength of such approaches lies in the effectiveness of the approach under operation.

A possible method to evaluate such an approach in operation are model-based testing as proposed by [6]. However, models always provide abstractions that may unintentionally mask some system properties so that the according results will not be able to match the real system in operation. Roske et al. specify the problem of testing and evaluation in more details in [8]. First of all, evaluation requires a safe environment because of lower tolerance in terms of unpredictable potential decision errors in real-life applications. Secondly, they propose a clear separation of concerns for testing in order to identify the potential source of inadequate system performance:

  1. 1.

    Testing the perception function: Perception is concerned with observing and sensing the environment and its characteristics. The data delivered by perception usually deals as the basis for decision making. Erroneous perception may lead to faulty decisions. The encapsulation of the perception function enables to fully control the hardware and the environment and allows to perform controlled failure injection.

  2. 2.

    Testing the decision making function: The decision making process is implemented according to an objective function and operates on the data delivered by the perception function. The evaluation of it mainly addresses to verify the implementation of the autonomous decisions (based on reliable perception data) and the question whether the system behavior approximates towards the objective function.

  3. 3.

    Testing the execution function: This is concerned with the testing ability to execute the decisions in order to obtain confidence about the system performance in terms of classical system functions (control performance, protection, reliability, etc.) as well as physical performance (speed, capacity, resource demand, etc.).

Evaluating the effectiveness of autonomous systems shall be explicitly concerned with testing against a specified set of requirements [12]. These are dictated by test missions set on the system performance (e.g. aspects of objective function) instead of testing the decision process based on assumptions made on decisions. For each test mission, the definition of test scenarios is essential to be combined with scenario parameters and metrics that quantify the satisfaction of the specified requirements.

Thompson suggests to use virtual environments to test autonomous software as they allow a system to be preliminarily executed in a safe environment preventing damage or harm of the target system and its environment. The most important requirement is that the virtual environment is identical to the real application environment in such a way that all essential physical properties and obstacles match the real-world ones. Additionally, the virtual environment must provide all necessary environment information and requires to be interaction-based in order to enable the target system to take all actions specified by its nature. To ensure comprehensive evaluation results, extraction of precise and suitable operation data from the virtual environment must be guaranteed.

2.3 Requirements

Evaluation of an autonomous approach can only be conducted within its application domain. The mission of our evaluation is to verify the anomaly detection approach performance according to its intended properties. Detecting changes in the applications, their parameters, in the environment and the system itself is the crucial task of the anomaly detection framework. Means to induce and control changes in the environment and the system are required to examine the detection quality in order to draw conclusions on the reliability of the detection mechanism. Observed behavior, especially previously unknown, has to be classified and declared either to be normal, suspicious or dangerous on the basis of system health signals provided by the OS. Failures may endanger the entire system and, consequently, shall be detected by the anomaly detection framework. In order to evaluate the classification part of the anomaly detection, unsafe or dangerous system state have to be initiated in a controlled manner by means to inject failures into the executing system. The following requirements on the evaluation environment and architecture can be formulated:

R.1 :

Reconfigurable Applications and Environment: Our evaluation target is designed to detect unintended and malicious system states provoked by autonomous decisions at the application side. Therefore, an application environment is required that is dynamically changing and executes self-reconfigurable applications as a basis to assess the effectiveness of our online anomaly detection.

R.2 :

Execution Platform: The validation of the anomaly detection is only possible in the context of a RTOS for autonomous environments.

R.3 :

Safe Environment: For executing the application, a safe environment is required to ensure the environment to be identical to the real one and to enable full control of the system entities. Full control in this context is related to e.g. controlled initiation of dynamical changes that lead to reconfigurations. Dynamical reconfigurations build up the basis to verify whether and to what extend these changes implicate the target approach in terms of detecting novel behaviors.

R.4 :

Separation of Evaluation Concerns: Based on the requirements formulated by Roske et al., evaluation environments shall make sure that clear system boundaries can be guaranteed. Considering the evaluation of the pure anomaly detection approach, the RTOS represents the evaluation environment. However, as the evaluation of this approach is strongly related to the application performance, the execution environment has to ensure these system boundaries as well.

R.5 :

Interaction-based Environment: Environmental changes are foundations of autonomous reactions. Therefore, interaction between virtual environment applied for evaluations and the associated application is essential.

R.6 :

Evaluation Output: We are mainly interested in whether the intended properties of The main entity of our evaluation is integrated into the OS which is responsible to deliver the results of the evaluation. However, the virtual application environment itself shall provide information to draw conclusions on the system performance from the according evaluation scenarios.

2.4 Applicability of Virtual Environments

VR enables to test mechatronic systems, even in early development phases. Especially, the VP inside a VE provides suitable instruments. For using virtual environments to evaluate such autonomous approaches as introduced above, the requirements imposed for the test environment (namely R.3 to R.6) have to be fulfilled.

VE and VP, by concept, are constructed based on real environment, real prototypes or devices (respective to e.g. the size, mass, physical behavior and so on). Hence, VE and the VP that operates inside act like their real counterparts and fulfills R.3. Using a VP inside a VE provides a tool to safely verify all aspects of the mechatronic system without endangering neither the real hardware nor the real environment. Every essential system entity of the real system is constructed independently as an encapsulated system entity in its virtual counterpart with all offered interfaces. By this, decoupled evaluations of individual system parts become possible and allows components to be exchanged by other implementations. This, in particular, forms the foundations for realizing the separation of concerns in the evaluation (see R.4).

VEs can provide interfaces for access to system entities and their parameters for analysis, assessment as well as manipulation purposes. Using these interfaces at run-time, changing system components, modifying the VE or even injecting failures into the system can be performed in an interactive manner according to requirement R.5.

VEs offer visual representation of the system under execution as well as visualizations of diverse parameters, including values of physical components and parameters of the executed software. Thereby, virtual environments provide facilities to visualize all parameters needed for the evaluation, as defined by requirement R.6.

3 Concept of the Virtual Test Environment

As foundation, we use a virtual test environment that has already been applied for the evaluation of other approaches [11]. This virtual test environment consists of a VP in forms of a mechatronic system which in our case is a miniature robot (that acts like its real counterparts, see R.3), and its VE being a randomly generated maze. The task of the robot is to find a way through the maze to a dedicated destination in an autonomous manner. The applied algorithm is implemented by an OS task executed on the VP. The randomly generated maze acts as a test environment for the applied algorithm.

In our virtual test environment, virtual (hardware) components and the executed RTOS with its tasks communicate with each other by dedicated interfaces. This enables decoupled evaluations of individual parts, in particular, the VE, the components of the VP, the execution part composed of RTOS and its tasks (and fulfills R.4) and, thereby, allows to exchange the execution of system parts, e.g. to execute the OS either on an emulator or on real hardware. R.5 requires changes on system components, modifications of the virtual environment and failure injection to be performed in an interactive manner. In our concept, we support the system user to implement such interactive behaviors by using a scripting language to specify own interaction methods quickly.

Our virtual test environment offers a visual representation of the scenario execution. Visualizations of diverse parameters, including values of physical components as well as parameters of the executed software, as required for evaluation by R.6, are provided by the implementation extensions made on the virtual test environment.

3.1 Evaluation Environment and Prototypical Implementation

Our virtual test environment is split into the virtual execution of application tasks with its OS, and the VE for the simulation of the VP hardware. Using the VP platform, early functional verification tests of newly implemented functions can be performed within its environment without having access to the real hardware platform.

The VE is connected to the OS execution platform by using a network connection which allows a seamless exchange between real and virtual hardware. Our approach proposes a stepwise refinement of the VE towards the real execution environment. In early phases, we are able to test all OS functionalities using a virtual execution platform in connection with our VE. In later phases, we are able to exchange either the VE with the real hardware or changing the virtual execution platform to the real hardware executing the OS. The interface to the real OS is abstracted away by an intermediate layer which will forward all system relevant function calls to the virtual platform.

By using a VE and a VP, we are able to change the behavior of both interactively. In early stages of developing we can disable all random factors like sensor noise or sensor latency for verifying the bare functionality of the algorithm. Later, noise and latency can be enabled to prove the algorithms functionality for the real hardware. Also, disabling specific hardware e.g. sensors to simulate an failure is offered. This helps verifying the fault tolerance of algorithms and respectively, the OS with its internal behavior.

Fig. 1.
figure 1

The virtual environment of BeBot and the interactive GUI.

Full size image

We choose the miniature robot BeBot [5] (Fig. 1) for our test scenario. BeBots are small robots serving as a technology platform. They are used for research in the domains of dynamic reconfigurable systems, multi agent systems and swarm intelligence. The dimensions of a BeBot are approximately \(9cm^3\) and it provides 12 IR sensors for sensing the environment. Robot movement is achieved by two motors which supports quasi omni-directional movements as it allows the BeBots to turn on the spot. The BeBot itself is controlled by ORCOS, a self-reconfigurable real-time operating system [2].

For the implementation of our virtual test environment and the BeBot as VP (see Fig. 1b) we used Unity3D, a professional game engine. To set up the VP we imported the BeBot CAD data used for production and added physical behavior. The maze is build up with virtual counterparts of brick-like plastic elements used for testing the real BeBot. The physical behavior of the BeBot is completely built using the internal physics engine of Unity3D. The IR sensors are realized as a physical ray sent out to the VE. If the ray hits a wall, it returns the distance where it intersects. For realism we added approx. \(10\,\%\) random noise to the virtual sensor value, which is the amount of jitter we measured on the real sensors.

On the software side, we try to get as near as possible to the real interface of the BeBot. Therefore, the control for the stepper motors, the sensors and the LEDs use the same messages and protocols we used on the real controller. With this approach, we are able to run the same code written for the BeBot platform either on a software or a hardware emulator or on the BeBot itself and combine it with the VP.

4 Implementation of the Evaluation

The evaluation target is the anomaly detection approach presented in Sect. 2.1 and was implemented and integrated into ORCOS, a self-reconfigurable RTOS. ORCOS runs on the BeBot that executes autonomously operating applications and therefore, was chosen as an application platform for evaluation of our target approach. The application running on the BeBot is reconfigurable (exchangeable strategies/algorithms) with respect to an objective function.

4.1 Evaluation Architecture

For evaluation, an architecture is required that combines the VE with its VP and the OS as the execution platform for the BeBot application. Figure 2a shows the architecture which contains the VE representing the environment (maze) and the VP (BeBot). For executing applications on BeBot, we use ORCOS that runs on the emulator QEMU. We have customized QEMU for ORCOS by enhancing the communication feature in forms of an Assistant Communication Module (ACM) that enables QEMU is connected to the VP. Once the virtual test environment is executing, ORCOS starts to communicate with the VP through ACM. The environment data provided by VE is transferred back to ORCOS. A particular BeBot controlling task processes these environment data and commands the virtual BeBot by sending messages to it. This entire architecture of the virtual test environment is implemented to on a Linux system.

Fig. 2.
figure 2

Scenario and Architecture overview.

Full size image

The goal of the BeBot in the VE is to reach the destination in the maze in an autonomous manner. We designed several maze-solving algorithms to control the BeBot in order to reach the destination efficiently:

  • Right-Side Wall Follower: The robot always follows the right-side wall in the maze and must ensure to not lose the wall it is following. For mazes that are not simply connected, this algorithm can not guarantee the reach the destination.

  • Left-Side Wall Follower: This algorithm uses the same principles as right-side wall follower by relying on the left-side instead of the right-side wall.

  • Random Mouse Algorithm: Whenever the robot meets a junction, the robot makes random decisions on which direction to follow. This algorithm does not guarantee that the robot will find the right solution. However, it can be applied in mazes in which not all the walls are connected and will probably find its destination.

  • Parameterized Random Mouse Algorithm: This algorithm is a mean to enhancement the probability of the robot to find its destination by using random mouse. The turning decision are randomized but weighed by parameters reflecting the distance calculated between the robot and its destination.

Each of the maze-solving algorithm is implemented as a single task that performs best in a specific scenario. Figure 2b specifies the application side architecture. The Task Controller implements the decision making part to determine the most suitable strategy for solving the maze according to its objective function. If the applied strategy is not suitable to the underlying maze, the Task Controller has to choose a better algorithm.

Behaviors of the task are recorded by the anomaly detection framework and analyzed in context of the corresponding OS health state to determine to classify the system behavior as normal or anomalous. If the anomaly detection framework detects any anomaly associated with one system part (task or OS component), e.g. an unreliable device, the anomaly detection framework forwards these information to the Task Controller in order initiate the Task Controller to choose another algorithm strategy from the task repository that meets the detected novel circumstances.

4.2 Environment Control and Interaction

For evaluation, full control over the VE and the VP is fundamental. First, we are able to set up the VE and the VP with respect to the evaluation requirements. The BeBot is the unit of the evaluation environment, that is part of the autonomous system and provides the interface to the VE. The BeBot’s interface to the VE are the IR Sensors. Our virtual test environment provides different abstraction levels for hardware devices that can be exchanged by the VE. Simulated real IR sensor data includes the transfer function, the latency and the noise measured in test series of the real IR sensors. Pure distance data represent the distance of the IR sensor to the detected wall in cm without any latency and noise. User-defined data the interactive control allows to configure the degree of the data interfered with latency and noise as well as the transfer function. E.g. it is possible to set up the distance values with noise added.

With these data abstraction levels, both, the perception function and the decision making process, can be evaluated in a finely graduated manner. Because the BeBot application’s decisions are expected to be different in different environmental configurations, in our VE, we can set up the actual maze as well as its type on the fly by using a randomized generation and by switching between different algorithms for maze construction. Furthermore, we are able to configure the start point of the BeBot and the destination position inside the maze.

Second, in our virtual test environment, full control at runtime is presumed and realized by integrating interactive control over VE and the BeBot. Not only the applied model of a device like the IR Sensor is configurable, furthermore, we allow to disable sensors completely in order to enable failure injection. The user can decide interactively between different options for defective sensor values send back to the OS in order to evaluate the reaction of the decision making process.

We have designed a GUI (Fig. 1a) to interactively change all parameters defined above. The great benefit from applying the virtual test environment is the possibility of controlling the VE and the hardware which involves the potentials for simulating failures in various ways. This flexibility offered by the VE cannot be provided by the real hardware or the real environment at the same level of simplicity.

4.3 Evaluation Scenarios

The objective of the evaluation process is to verify the performance and the effectiveness of the autonomous system. We have to deal with autonomous behavior on two different levels: the first is the anomaly detection approach that in fact is our evaluation target, and the second is the BeBot application with its objective to autonomously find a way to a predefined destination in a maze. The autonomous BeBot application builds up the basis (referring to requirement R.1) for the evaluation of our anomaly detection framework. Only with a reliable application environment, we are able to examine, analyze and evaluate the performance of the anomaly detection framework. The two different levels in autonomous behavior cause different views on separation of evaluation concerns as described in R.4. Considering the BeBot application first, we clearly define system boundaries for the individual system parts.

BeBot Application’s Perception Function: The BeBot’s perception function is sensing the environment by data delivered from its IR sensors in order to measure the distances to obstacles. Data sensing functions are clearly separated components in the VE and allow separated evaluations. The different abstraction levels determine the accuracy of the perception data and allow to evaluate to what extent the implemented data sensing module matches the reality. In addition, full control over the reliability of the data delivered to the decision making module can be ensured.

BeBot Application Decision Function: In the BeBot application, the decision function is implemented in the software in forms of the task and the Task Controller. While the task is executing on the basis of the perception data, the Task Controller examines the performance of the task. It is responsible for the decision to either maintain the executing strategy or to reconfigure the application and exchange it by another strategy. A reconfiguration is intended in case of faulty hardware (such as a sensor) or in case the application underperforms the objective function. The decision making process can only be evaluated in separate, if the reliability of the sensor data can be guaranteed. We define test scenarios determined by missions which exhaust and evaluate the specifications set in our decision making process.

BeBot Application Execution Function: The execution function is defined by movement of the BeBot within its environment. This part of the evaluation concentrates on testing the effectiveness of the implemented application strategies and aims to show that the BeBot is able to find the destination. This includes also reconfiguration of the application if configured that did not fulfill the objective.

With these evaluation missions, we are able to evaluate the different parts of the performance of the BeBot application. Even though, we are more interested in the evaluation of performance of the anomaly detection integrated into ORCOS. Now, we examine the system boundaries attached to the evaluation of the anomaly detection approach and verify whether these are realized within our virtual test environment.

Anomaly Detection Perception Function: Our anomaly detection relies on perception of the application behavior in combination with monitored OS health state. The application behavior observation implemented by a System Call Monitor integrated into ORCOS that has been evaluated separately in [13]. The OS health state is examined by an OS Health Monitor [7] which deal as system-wide input signals indicating the respective heal state of the RTOS for the context-related classification of our anomaly detection approach. Parameters such as processor utilization, memory usage, etc. are examined, but also device drivers which involves the data obtained from the sensors of the VP. IR sensor data represents the perception function of the BeBot device or respectively, its virtual counterpart. Erroneous sensors or erroneous data are intended to identified by condition scenarios specified by the OS Health Monitor and are revealed through the system-wide signals. In order to guarantee correctness of the perception function, the performance of the OS Health Monitor including the analysis of the IR sensor data has been evaluated in [7].

Anomaly Detection Decision Function: The decision making function in the anomaly detection approach is defined by the classification mechanism for behavioral patterns. It can only be evaluated when variations in system behavior are induced. Here, the functionality of virtual test environment offers great potentials, as it allows to induce dynamical changes into the VP and the VE as well (see Sect. 4.2). Changes in e.g. sensor behavior shall lead to changes in classification outcomes of the anomaly detection approach and in turn may effect reconfiguration initiations. One main objective is the verification of the ability to detect behavioral changes and suspicious OS health state. Related to this, accurate configurations of decision thresholds for detection as well as for classification are examined.

Anomaly Detection Execution Function: This part of evaluation concentrated on the question whether in case of detected dangerous behavior, the information is passed to the Controller Task in order to initiate an adequate system reconfiguration.

Each of these separate functions is evaluated within our virtual test environment individually by defined scenarios. The scenarios aim to test only the function concerned while assuming correct functionality of the remaining system parts.

4.4 Output

The anomaly detection operates encapsulated inside ORCOS on the basis of the data delivered by the VE. Its evaluation output is encoded within the RTOS by reporting the results in forms of log files that have to be analyzed after execution. However, decision results directly affect the BeBots performance as a task reconfiguration might have been induced. The BeBot application is executed in the virtual test environment which provides evaluation output related to that. All required information about the performance of the system must be able to be acquired in the VE: With the interaction interface as introduced, we get control over the system entities, but furthermore, this interface reports all information about the state of the system entities at runtime. While the BeBot is driving through the maze, the BeBot application controls the action of the BeBot as well as the performance of the application itself. All decisions made by the application, either BeBot controlling commands or application configuration are displayed at runtime in the VE. Applying the VE for evaluation not only enables to execute the real application within a safe environment in order to asses its performance. Furthermore, it visualizes the performance of the application, so that the developer can directly observe the execution and immediately recognize potential problems. All aspects of the evaluation process are supported by the virtual test environment. The virtual test environment continuously illustrates all evaluation aspects and performance metrics at runtime. Furthermore, what in particular is specific to VE, it illustrates the current execution process with all the associated data in a visual manner. This visualization makes it easier to understand and follow the execution and capture the performance of the autonomous application in order to assess it.

5 Conclusion

In this paper, we address the problem of evaluating mechatronic systems that exhibit autonomous behavior. For these novel research approaches, technologies are required that meet the requirements associated with their evaluation. VEs offer great potentials to satisfy these requirements. The evaluation target presented in this paper is an autonomous anomaly detection framework for self-reconfiguring real-time operating systems. We formulated specific requirements concerning the evaluation of this approach. The evaluation target as encapsulated in an operating system requires an evaluation environment with a concrete (autonomous) application. Virtual reality, in particular VEs and VPs, provide safe environments and full control over all parameters and thereby makes it powerful to be applied for evaluation purposes. We developed a virtual test environment consisting of a VP, an application, controlling the VP, which runs on an emulator that executes our operating system with the anomaly detection. This architecture defines clear system boundaries required for evaluation. The design of this virtual test environment meets all the requirements specified for evaluating autonomous mechatronic systems. Furthermore, the presented virtual test environment allows to interactively manipulate and control all evaluation aspects that are defined by evaluation scenarios. With VEs for evaluation generating variations of evaluation application environments for autonomous mechatronic systems can be done with little effort, therefore VEs will become a general-purpose evaluation platform.