Adaptive cruise control via adaptive dynamic programming with experience replay

Abstract

The adaptive cruise control (ACC) problem can be transformed to an optimal tracking control problem for complex nonlinear systems. In this paper, a novel highly efficient model-free adaptive dynamic programming (ADP) approach with experience replay technology is proposed to design the ACC controller. Experience replay increases the data efficiency by recording the available driving data and repeatedly presenting them to the learning procedure of the acceleration controller in the ACC system. The learning framework that combines ADP with experience replay is described in detail. The distinguishing feature of the algorithm is that when estimating parameters of the critic network and the actor network with gradient rules, the gradients of historical data and current data are used to update parameters concurrently. It is proved with Lyapunov theory that the weight estimation errors of the actor network and the critic network are uniformly ultimately bounded under the novel weight update rules. The learning performance of the ACC controller implemented by this ADP algorithm is clearly demonstrated that experience replay can increase data efficiency significantly, and the approximate optimality and adaptability of the learned control policy are tested with typical driving scenarios.

Appendices

Appendices

A Proof of Lemma 1

The first difference of $L_1(t)$ is

$$\begin{aligned} \Delta L_1(t)=\,&\frac{\lambda _1}{\beta }tr\left[ \tilde{w}^T_\mathrm{c}(t+1) \tilde{w}_\mathrm{c}(t+1)-\tilde{w}^T_\mathrm{c}(t)\tilde{w}_\mathrm{c}(t)\right] \nonumber \\ =\,&\frac{\lambda _1}{\beta }tr\left[ (\tilde{w}_\mathrm{c}(t) \right. +\Delta \hat{w}_\mathrm{c}(t))^T(\tilde{w}_\mathrm{c}(t)+\Delta \hat{w}_\mathrm{c}(t))\nonumber \\&-\,\left. \tilde{w}^T_\mathrm{c}(t)\tilde{w}_\mathrm{c}(t)\right] \end{aligned}$$

(25)

with the basic property of matrix $tr(AB)=tr(BA)$, then

$$\begin{aligned} \Delta L_1(t)=\frac{\lambda _1}{\beta }tr\left[ 2\tilde{w}^T_\mathrm{c}(t)\Delta \hat{w}_\mathrm{c}(t)+\Delta \hat{w}^T_\mathrm{c}(t)\Delta \hat{w}_\mathrm{c}(t)\right] \end{aligned}$$

(26)

Substituting (9) into (26), we obtain

$$\begin{aligned} \Delta L_1(t)=\,&\lambda _1tr\left[ -2\gamma \tilde{w}_\mathrm{c}^T(t)\phi _\mathrm{c}(t)e_\mathrm{c}^T(t)\right. \nonumber \\&-\,2\gamma \tilde{w}_\mathrm{c}^T(t) \sum _{j=1}^{l}\phi _{c_j}(t)e_{c_j}^T(t) \nonumber \\&+\,\beta \gamma ^2\left( \phi _\mathrm{c}(t)e_\mathrm{c}^T(t)+\sum _{j=1}^{l} \phi _{c_j}(t)e_{c_j}^T(t)\right) ^T \nonumber \\&\times \left. \left( \phi _\mathrm{c}(t)e_\mathrm{c}^T(t)+\sum _{j=1}^{l} \phi _{c_j}(t)e_{c_j}^T(t)\right) \right] \nonumber \\ =&\lambda _1tr\left[ \mathfrak {R}_1(t)+\mathfrak {R}_2(t)\right] \end{aligned}$$

(27)

where

$$\begin{aligned} \mathfrak {R}_1(t)=\,&-2\gamma \tilde{w}_\mathrm{c}^T(t)\phi _\mathrm{c}(t)e_\mathrm{c}^T(t)-2\gamma \tilde{w}_\mathrm{c}^T(t) \sum _{j=1}^{l}\phi _{c_j}(t)e_{c_j}^T(t)\\ \mathfrak {R}_2(t)=&\beta \gamma ^2\left( \phi _\mathrm{c}(t)e_\mathrm{c}^T(t) +\sum _{j=1}^{l}\phi _{c_j}(t)e_{c_j}^T(t)\right) ^T\\&\times \left( \phi _\mathrm{c}(t)e_\mathrm{c}^T(t)+\sum _{j=1}^{l}\phi _{c_j}(t)e_{c_j}^T(t)\right) \end{aligned}$$

with (7) we have

$$\begin{aligned} e_\mathrm{c}(t)=\,&\gamma \hat{w}_\mathrm{c}^T(t)\phi _\mathrm{c}(t)+r(t)-\hat{w}_\mathrm{c}^T(t-1)\phi _\mathrm{c}(t-1)\nonumber \\ =&\gamma \xi _\mathrm{c}(t)+\gamma w_\mathrm{c}^T(t)\phi _\mathrm{c}(t)+r(t)-\hat{w}_\mathrm{c}^T(t-1)\phi _\mathrm{c}(t-1) \end{aligned}$$

(28)

For simplicity, we remark

$$\begin{aligned} P(t)=\gamma w^{T}_\mathrm{c}\phi _\mathrm{c}(t)+r(t)-\hat{w}^T_\mathrm{c}(t-1)\phi _\mathrm{c}(t-1) \end{aligned}$$

(29)

then

$$\begin{aligned} \mathfrak {R}_1(t)=\,&-\,2\gamma \xi _\mathrm{c}(t)\left( \gamma \xi _\mathrm{c}(t)+P(t)\right) ^T\\&-\,2\gamma \tilde{w}_\mathrm{c}^T(t)\sum _{j=1}^{l}\phi _{c_j}(t)e_{c_j}^T(t) \end{aligned}$$

$$\begin{aligned} \mathfrak {R}_2(t)=\,&\beta \gamma ^2\Vert \phi _\mathrm{c}(t)\Vert ^2 \left( \gamma \xi _\mathrm{c}(t)+P(t)\right) \left( \gamma \xi _\mathrm{c}(t)+P(t)\right) ^T\\&+\,2\beta \gamma ^2\phi _\mathrm{c}(t)(\gamma \xi _\mathrm{c}(t)+P(t))^T\sum _{j=1}^{l} \phi _{c_j}(t)e_{c_j}^T(t)\\&+\,\beta \gamma ^2\bigg \Vert \sum _{j=1}^{l}\phi _{c_j}(t)e_{c_j}^T(t)\bigg \Vert ^2 \end{aligned}$$

Substituting $\mathfrak {R}_1(t)$ and $\mathfrak {R}_2(t)$ into (27), we obtain

$$\begin{aligned} \Delta L_1(t)=\,&-\lambda _1\gamma ^2\left( 1-\beta \gamma ^2\Vert \phi _\mathrm{c}(t)\Vert ^2\right) \Vert \xi _\mathrm{c}(t)\Vert ^2\nonumber \\&-\,\lambda _1\left( 1-\beta \gamma ^2\Vert \phi _\mathrm{c}(t)\Vert ^2\right) \Vert \gamma \xi _\mathrm{c}(t)+P(t)\Vert ^2\nonumber \\&-\,\lambda _1\gamma ^2\Vert \tilde{w}_\mathrm{c}(t)\Vert ^2-\lambda _1 \left( 1-\beta \gamma ^2\Vert \phi _\mathrm{c}(t)\Vert ^2\right) \nonumber \\&\times \bigg \Vert \sum _{j=1}^{l}\phi _{c_j}(t)e_{c_j}^T(t)\bigg \Vert ^2 -\lambda _1\beta \gamma ^2\Bigg \Vert \gamma \xi _\mathrm{c}(t)\phi _\mathrm{c}^T(t)\nonumber \\&-\,\sum _{j=1}^{l}\phi _{c_j}(t)e_{c_j}^T(t)\Bigg \Vert ^2 +\lambda _1\left( 1-\beta \gamma ^2\Vert \phi _\mathrm{c}(t)\Vert ^2\right) \nonumber \\&\times \Vert P(t)\Vert ^2 +\lambda _1\bigg \Vert \gamma \tilde{w}_\mathrm{c}^T(t) -\sum _{j=1}^{l}\phi _{c_j}(t)e_{c_j}^T(t) \bigg \Vert ^2\nonumber \\&+\,\lambda \beta \gamma ^2\bigg \Vert P(t)\phi _\mathrm{c}^T(t)-\sum _{j=1}^{l} \phi _{c_j}(t)e_{c_j}^T(t) \bigg \Vert ^2 \end{aligned}$$

(30)

Applying the Cauchy–Schwarz inequality $\Vert x+y\Vert ^2\le 2\Vert x\Vert ^2+2\Vert y\Vert ^2$, we can derive (20).

The proof is completed.

B Proof of Lemma 2

The first difference of $L_2(t)$ is

$$\begin{aligned} \Delta L_2(t)=&\frac{\lambda _2\gamma ^2}{\alpha }tr \left[ \tilde{w}^T_\mathrm{a}(t+1) \tilde{w}_\mathrm{a}(t+1)-\tilde{w}^T_\mathrm{a}(t)\tilde{w}_\mathrm{a}(t)\right] \nonumber \\ =&\frac{\lambda _2\gamma ^2}{\alpha }tr\left[ (\tilde{w}_\mathrm{a}(t) +\Delta \hat{w}_\mathrm{a}(t))^T(\tilde{w}_\mathrm{a}(t)+\Delta \hat{w}_\mathrm{a}(t))\right. \nonumber \\&-\,\left. \tilde{w}^T_\mathrm{a}(t)\tilde{w}_\mathrm{a}(t)\right] \nonumber \\ =&\frac{\lambda _2\gamma ^2}{\alpha }tr\left[ 2\tilde{w}^T_\mathrm{a}(t) \Delta \hat{w}_\mathrm{a}(t)+\Delta \hat{w}^{T}_\mathrm{a}(t)\Delta \hat{w}_\mathrm{a}(t)\right] \end{aligned}$$

(31)

Substituting (15) into (31), we have

$$\begin{aligned} \Delta L_2(t)=&\lambda _2\gamma ^2\left[ \bigg \Vert \xi _\mathrm{a}(t)C(t) -\hat{w}_\mathrm{c}^T(t)\phi _\mathrm{c}(t)\bigg \Vert ^2\right. \nonumber \\&-\,\Vert \hat{w}_\mathrm{c}^T(t)\phi _\mathrm{c}(t)\Vert ^2-\Vert C(t)\Vert ^2\Vert \xi _\mathrm{a}(t)\Vert ^2+\Vert \tilde{w}_\mathrm{a}(t)\Vert ^2\nonumber \\&-\,\bigg \Vert \tilde{w}_\mathrm{a}^T(t)+\sum _{j=1}^{l} \phi _{a_j}(t)C_j(t)e_{a_j}^T(t)\bigg \Vert ^2\nonumber \\&+\,\bigg \Vert \sum _{j=1}^{l}\phi _{a_j}(t)C_j(t)e_{a_j}^T(t)\bigg \Vert ^2\nonumber \\&+\,\left. \alpha \bigg \Vert \phi _\mathrm{a}(t)C(t)e_\mathrm{a}^T(t){+}\sum _{j=1}^{l} \phi _{a_j}(t)C_j(t)e_{a_j}^T(t)\bigg \Vert ^2\right] \end{aligned}$$

(32)

By Cauchy–Schwarz inequality, we can derive (22).

The proof is completed.

C Proof of Theorem 1

Consider the following Lyapunov function candidate

$$\begin{aligned} L(t)=L_1(t)+L_2(t) \end{aligned}$$

where

$$\begin{aligned} L_1(t)=&\frac{\lambda _1}{\beta }tr[\tilde{w}^T_\mathrm{c}(t)\tilde{w}_\mathrm{c}(t)]\\ L_2(t)=&\frac{\lambda _2}{\alpha }tr[\tilde{w}^T_\mathrm{a}(t)\tilde{w}_\mathrm{a}(t)] \end{aligned}$$

The first difference of L(t) is

$$\begin{aligned} \Delta L(t)=&L(t+1)-L(t)=\Delta L_1(t)+\Delta L_2(t) \end{aligned}$$

(33)

By employing Lemma 1 and Lemma 2, we obtain

$$\begin{aligned} \Delta L(t) \le&-\gamma ^2\left[ \lambda _1\left( 1-\beta \gamma ^2\Vert \phi _\mathrm{c}(t)\Vert ^2\right) -2\lambda _2\right] \Vert \xi _\mathrm{c}(t)\Vert ^2\nonumber \\&-\,\lambda _1\left( 1-\beta \gamma ^2\Vert \phi _\mathrm{c}(t)\Vert ^2\right) \bigg \Vert \gamma \xi ^T_\mathrm{c}(t)+\gamma w^{T}_\mathrm{c}(t)\phi _\mathrm{c}(t)\nonumber \\&+\,r(t)-\hat{w}^T_\mathrm{c}(t-1)\phi _\mathrm{c}(t-1)\bigg \Vert ^2\nonumber \\&-\,\lambda _1\beta \gamma ^2\bigg \Vert \gamma \phi _\mathrm{c}(t)\xi _\mathrm{c}^T(t) -\sum _{j=1}^{l}\phi _{c_j}(t)e_{c_j}^T(t)\bigg \Vert ^2\nonumber \\&-\,\lambda _2\gamma ^2\Vert C(t)\Vert ^2\Vert \xi _\mathrm{a}(t)\Vert ^2\nonumber \\&-\,\lambda _2\gamma ^2\bigg \Vert \tilde{w}_\mathrm{a}^T(t)+\sum _{j=1}^{l} \phi _{a_j}(t)C_j(t)e_{a_j}^T(t)\bigg \Vert ^2\nonumber \\&-\,2\lambda _2\gamma ^2\left( 1-\alpha \Vert C(t)\Vert ^2\Vert \phi _\mathrm{a}(t)\Vert ^2\right) \bigg \Vert \hat{w}_\mathrm{c}^T(t)\phi _\mathrm{c}(t)\bigg \Vert ^2\nonumber \\&+\,\mathfrak {L}^2(t) \end{aligned}$$

(34)

where

$$\begin{aligned} \mathfrak {L}^2(t)=&\, \lambda _1\left( 1+\beta \gamma ^2\Vert \phi _\mathrm{c}(t)\Vert ^2\right) \bigg \Vert \gamma w^{T}_\mathrm{c}(t)\phi _\mathrm{c}(t)+r(t)\nonumber \\&-\,\hat{w}^T_\mathrm{c}(t-1)\phi _\mathrm{c}(t-1)\bigg \Vert ^2\nonumber \\&+\,\lambda _1\left( 1+3\beta \gamma ^2\right) \bigg \Vert \sum _{j=1}^{l}\phi _{c_j}(t)e_{c_j}^T(t)\bigg \Vert ^2\nonumber \\&+\,\lambda _1\gamma ^2\Vert \tilde{w}_\mathrm{c}(t)\Vert ^2 +\lambda _2\gamma ^2\Vert \tilde{w}_\mathrm{a}(t)\Vert ^2\nonumber \\&+\,2\lambda _2\gamma ^2\Vert w_\mathrm{c}^T\phi _\mathrm{c}(t)\Vert ^2\nonumber \\&+\,\lambda _2\gamma ^2\bigg \Vert \xi _\mathrm{a}(t)C(t)-\hat{w}_\mathrm{c}^T(t)\phi _\mathrm{c}(t)\bigg \Vert ^2\nonumber \\&+\,\lambda _2\gamma ^2(1+2\alpha )\bigg \Vert \sum _{j=1}^{l}\phi _{a_j} (t)C_j(t)e_{a_j}^T(t)\bigg \Vert ^2 \end{aligned}$$

(35)

With Assumption 1 and Cauchy–Schwarz inequality, we derive from (35)

$$\begin{aligned} \mathfrak {L}^2(t)\le&\, 3\lambda _1\left[ 2l^2+\beta \gamma ^2(1+6l^2 +\gamma ^2)\right] w_\mathrm{cm}^2\phi _\mathrm{cm}^4\nonumber \\&+\,\left[ 3\lambda _1+\gamma ^2(3\lambda _1+4\lambda _2)\right. \nonumber \\&+\,\left. \lambda _2\gamma ^{2}l^2(1+2\alpha )\phi _\mathrm{am}^2C_m^2\right] w_\mathrm{cm}^2\phi _\mathrm{cm}^2\nonumber \\&+\,3\lambda _1\left[ 1+l^2+\beta \gamma ^2(\phi _\mathrm{cm}^2+9l^2)\right] r_m^2 +\lambda _1\gamma ^2w_\mathrm{cm}^2\nonumber \\&+\,\gamma ^2(\lambda _1+2\lambda _2C_m^2\phi _\mathrm{am}^2)w_\mathrm{am}^2\triangleq \mathfrak {L}_m^2 \end{aligned}$$

(36)

where $r_m$ and $C_m$ are the upper bound of $\Vert r(t)\Vert$ and $\Vert C(t)\Vert$, respectively, that is $\Vert r\Vert \le r_m$, $\Vert C(t)\Vert \le C_m$.

Select the parameters to satisfy that

$$\begin{aligned} 0<\alpha \Vert C(t)\Vert ^2\Vert \phi _\mathrm{a}(t)\Vert ^2<1,\quad 0<\beta \gamma ^2\Vert \phi _\mathrm{c}(t)\Vert ^2<1 \end{aligned}$$

(37)

and the parameters $\lambda _i>0\ (i=1,2)$ are chosen as

$$\begin{aligned} \frac{\lambda _1}{\lambda _2}>\frac{2}{1-\beta \gamma ^2\Vert \phi _\mathrm{c}(t)\Vert ^2} \end{aligned}$$

(38)

Then, if (37) and (38) hold, for any

$$\begin{aligned} \Vert \xi _\mathrm{c}(t)\Vert >&\sqrt{\frac{\mathfrak {L}_m^2}{\gamma ^2\left[ \lambda _1 \left( 1-\beta \gamma ^2\Vert \phi _\mathrm{c}(t)\Vert ^2\right) -2\lambda _2\right] }} \end{aligned}$$

(39)

$$\begin{aligned} \Vert \xi _\mathrm{a}(t)\Vert >&\sqrt{\frac{\mathfrak {L}_m^2}{\lambda _2\gamma ^2\Vert C(t)\Vert ^2}} \end{aligned}$$

(40)

the first difference $\Delta L(t)<0$ holds.

Note that $\Vert \xi _\mathrm{c}(t)\Vert \le \Vert \tilde{w}_\mathrm{c}(t)\Vert \Vert \phi _\mathrm{cm}\Vert$ and $\Vert \xi _\mathrm{a}(t)\Vert \le \Vert \tilde{w}_\mathrm{a}(t)\Vert \Vert \phi _\mathrm{am}\Vert$, then by (39) and (40), we have

$$\begin{aligned} \Vert \tilde{w}_\mathrm{c}(t)\Vert>&\frac{1}{\phi _\mathrm{cm}}\sqrt{\frac{\mathfrak {L}_m^2}{\gamma ^2\left[ \lambda _1\left( 1-\beta \gamma ^2\Vert \phi _\mathrm{c}(t)\Vert ^2\right) -2\lambda _2\right] }}\triangleq \mathfrak {B}_\mathrm{c}\\ \Vert \tilde{w}_\mathrm{a}(t)\Vert >&\frac{1}{\phi _\mathrm{am}} \sqrt{\frac{\mathfrak {L}_m^2}{\lambda _2\gamma ^2\Vert C(t)\Vert ^2}}\triangleq \mathfrak {B}_\mathrm{a} \end{aligned}$$

With the standard Lyapunov extension theorem, we can conclude that the weight estimation errors of the critic network $\tilde{w}_\mathrm{c}(t)$ and the actor network $\tilde{w}_\mathrm{a}(t)$ are UUB by positive constants $\mathfrak {B}_\mathrm{c}$ and $\mathfrak {B}_\mathrm{a}$.

The proof is completed.