Formal Verification of Intelligent Hybrid Systems that are Modeled with Simulink and the Reinforcement Learning Toolbox

Abstract

Reinforcement Learning (RL) is a powerful technique to control autonomous hybrid systems (HSs) in dynamic and uncertain environments but makes it hard to guarantee their correct behavior in safety-critical applications. To formally guarantee safe behavior, a formal system description is required, which is often not available in industrial design processes and hard to obtain for the unpredictable, trial and error learning processes of RL. In this paper, we present an approach for semi-automatic deductive verification of intelligent HSs with embedded RL components modeled in Simulink together with the RL Toolbox. Our key ideas are threefold: First, we capture the safety-relevant behavior of RL components with hybrid contracts in differential dynamic logic. Second, we verify safety properties of the overall system with the RL component replaced by its contract deductively using the interactive theorem prover KeYmaera X. To make this possible, we precisely capture the semantics of industrially designed intelligent HSs by extending an existing transformation from Simulink to differential dynamic logic to support RL components. Third, we ensure that contracts are complied with at runtime by automatically deriving runtime monitors from our hybrid contracts. We demonstrate the practical applicability, scalability, and flexibility of our approach by verifying collision freedom of an autonomous intelligent robot in a factory setting.