Collision-Free Trajectory Generation and Tracking for UAVs Using Markov Decision Process in a Cluttered Environment

Abstract

A collision-free trajectory generation and tracking method capable of re-planning unmanned aerial vehicle (UAV) trajectories can increase flight safety and decrease the possibility of mission failures. In this paper, a Markov decision process (MDP) based algorithm combined with backtracking method is presented to create a safe trajectory in the case of hostile environments. Subsequently, a differential flatness method is adopted to smooth the profile of the rerouted trajectory for satisfying the UAV physical constraints. Lastly, a flight controller based on passivity-based control (PBC) is designed to maintain UAV’s stability and trajectory tracking performance. Simulation results demonstrate that the UAV with the proposed strategy is capable of avoiding obstacles in a hostile environment.

Appendices

Appendix A

Based on the condition that slide angle β = 0 and pitch angle 𝜃 = 0, the partial differential of the Rayleigh dissipation function ($\frac {\partial F}{\partial \dot {\phi } }, \frac {\partial F}{\partial \dot {\varphi } })$ are as follows:

$$\begin{array}{@{}rcl@{}} \frac{\partial F}{\partial \dot{\phi} }\!&=&\!-1051.05\cos \left( \varphi \right)\left( 6.6(-11\dot{\phi} \,+\,5\dot{\varphi} \cos \left( \phi \right) \right)\sin \left( \phi \right)/V)V^{2}\\ &&\!+ 1051.05\cos \left( \phi \right)\sin {\left( \varphi \right)(-0.4461-92.4424}\dot{\varphi} \sin \left( \phi \right)/V\\ &&\!+ 0.11(1.0656\,+\,24.6016\left( \frac{\dot{\varphi} \sin {\left( \phi \right))}}{V}\,+\,0.143V^{2} \right)\\ &&\!-1051.05(\sin {\left( \phi \right)\sin {\left( \varphi \right))\!\left( 6.6\left( 1.7\dot{\phi} \,-\,11.5\varphi \cos \left( \phi \right) \right)\!/V \right)}}V^{2}\\ \frac{\partial F}{\partial \dot{\varphi} }&=&-1051.05\cos \left( \varphi \right)\left( 6.6(-11\dot{\phi} + 5\dot{\varphi} \cos \left( \phi \right) \right)/V)V^{2}\\ &&-1051.05\sin {\left( \phi \right)(-0.4461-92.4424}\dot{\varphi} \sin \left( \phi \right)\cos \left( \theta \right)/V\\ &&+ 0.11(1.0656 + 24.6016\left( \frac{\dot{\varphi} \sin {\left( \phi \right))}}{V}+ 0.143V^{2} \right)\\ &&-1051.05\cos \left( \phi \right)\left( \frac{6.6\left( -11.5\dot{\varphi} \cos \left( \phi \right) \right)}{V} \right)V^{2} \end{array} $$

Input matrix $M=\left [ {\begin {array}{*{20}c} M_{\phi ,\delta _{a}} & M_{\phi ,\delta _{r}}\\ M_{\varphi ,\delta _{a}} & M_{\varphi ,\delta _{r}} \end {array}} \right ]$ which are due to drag, lift and side force can be expressed as follows:

$$\begin{array}{@{}rcl@{}} M_{\phi,\delta_{a}}&=&-630.63V^{2}\cos \left( \varphi \right)\\ M_{\phi,\delta_{r}}&=&231.231V^{2}\cos \left( \varphi \right)-634.413V^{2}\sin {\left( \phi \right)\sin \left( \varphi \right)}\\ M_{\varphi,\delta_{a}}&=&0\\ M_{\varphi,\delta_{r}}&=&-634.413V^{2}\cos \left( \phi \right) \end{array} $$

Appendix B

• A. Procedure of Policy Iteration

  • 1. Initialization

    V (s) ∈ R and π(s) ∈ A(s) arbitrarily for all s ∈ S

  • 2. Policy Evaluation Repeat 𝜗←0 For each s ∈ S:

    $$v\leftarrow V(s) $$
    $$V(s)\leftarrow \sum\limits_{s^{\prime}} {P_{ss^{\prime}}^{a}\left[ R_{ss^{\prime}}^{a}+\gamma V(s^{\prime}) \right]} $$
    $$\vartheta \longleftarrow max\left( \vartheta, \left| v-V(s) \right| \right) $$

    until 𝜗 < 𝜃 (a small positive number)

  • 3. Policy Improvement

    Policy stable ← true

    For each s ∈ S:

    $$ b\leftarrow \pi (s)$$
    $$\pi (s)\leftarrow arg ~{max}_{a}\sum\limits_{s\prime} {P_{ss^{\prime}}^{a}\left[ R_{ss^{\prime}}^{a}+\gamma V(s^{\prime}) \right]} $$

    If b≠π(s), then Policy stable ← false

    If policy stable, then stop; else, go to 2.

Appendix C

Fig. 16

The integration of backtracking method with MDP