Reinforcement Learning Approach to Generate Zero-Dynamics Attacks on Control Systems Without State Space Models

Abstract

Stealthy attacks on control systems are bound to go unnoticed, which makes them a severe threat to critical infrastructure such as power systems, smart grids, and vehicular networks. This paper investigates a subset of stealthy attacks known as zero-dynamics-based stealthy attacks. While previous works on zero-dynamics attacks have highlighted the necessity of highly accurate knowledge of the system’s state space for generating attack signals, our study requires none. We propose a deep reinforcement learning based attacker to generate attack signals without prior knowledge of the system’s state space. We develop several attackers and detectors iteratively until the attacker and detectors no longer improve. In addition, we also show that the reinforcement learning based attacker successfully executes an attack in the same manner as the theoretical attacker described in previous literature.

Appendices

A Proof for Attack Generation

Since matrix D is 0, we replace the solution in [2] as shown below. We need a value of $F_2$ that satisfies

From this, we can easily obtain

where, $\begin{bmatrix} V & B \end{bmatrix} ^ +$ and $V^+$ represent the Moore-Penrose pseudo-inverses of matrices $\begin{bmatrix} V & B \end{bmatrix}$ and V, respectively.

If we choose $\begin{bmatrix} F_1\\ F_2 \end{bmatrix}$ to be the right-hand side of (8b), that is,

$$\begin{aligned} \begin{bmatrix} F_1\\ F_2 \end{bmatrix}= \begin{bmatrix} V & B \end{bmatrix}^+ AVV^+, \end{aligned}$$

(9)

then using one of the properties of the pseudo-inverse, that states that for a general matrix M we have $M^+MM^+=M^+$, we see that this choice of $\begin{bmatrix}F_1\\ F_2\end{bmatrix}$ satisfies (8b). Now substituting the same in the left-hand side of (8a), and using the property that for a general matrix M we have that $MM^+M=M$, we get

$$\begin{aligned} \begin{bmatrix} V B \end{bmatrix} \begin{bmatrix} F_1\\ F_2 \end{bmatrix}V= \begin{bmatrix} V & B \end{bmatrix} \begin{bmatrix} V & B \end{bmatrix}^+ AV, \end{aligned}$$

(10)

the right-hand side of which is equal to AV (meaning that our choice of $\begin{bmatrix}F_1\\ F_2\end{bmatrix}$ satisfies (8a)) whenever $\begin{bmatrix} V B\end{bmatrix}$ has linearly independent rows. What remains is to set $F=-F_2$.

B Architectures and Hyperparameters

Table 4 shows the architecture of neural network models associated with the attackers. The table displays the number of nodes in each subsequent layers of the model, activation function in hidden and output layers, and learning rate and optimizer used to update the model’s parameters. Similarly, the Table 5 consists of hyperparameter settings for the neural network models. The target network is updated using a soft update method whose coefficient is 0.005 for each attacker. Similarly, the output of these attackers is scaled using the Tanh activation function. Hence, to get the desired attack signals, we scaled the output by some constant, depicted in the same table. In addition to this, the value of the parameters a and b from the reward function in Sect. 4.2 for each attacker are provided in the table. In addition, Table 6 consists of architectures and hyperparameters of LSTM detectors. All the detectors have the same architecture but differ in learning rate parameter, which is given in the table.

Table 4. Architecture of the neural network models associated with the attackers. Attackers ($ATT_1$, $ATT_2$) and ($ATT_3$, $ATT_4$ and $ATT_5$) share the same architecture

Table 5. Hyper Parameter Setting for the attackers based on SAC algorithm

Table 6. Architecture of the LSTM-based detectors

C Attacker Rewards and Performance

Fig. 4.

Reward plots of five consecutive attackers

Fig. 5.

Cosine similarity as described in Sect. 3.2.