A tutorial on value function approximation for stochastic and dynamic transportation

Abstract

This paper provides an introductory tutorial on Value Function Approximation (VFA), a solution class from Approximate Dynamic Programming. VFA describes a heuristic way for solving sequential decision processes like a Markov Decision Process. Real-world problems in supply chain management (and beyond) containing dynamic and stochastic elements might be modeled as such processes, but large-scale instances are intractable to be solved to optimality by enumeration due to the curses of dimensionality. VFA can be a proper method for these cases and this tutorial is designed to ease its use in research, practice, and education. For this, the tutorial describes VFA in the context of stochastic and dynamic transportation and makes three main contributions. First, it gives a concise theoretical overview of VFA’s fundamental concepts, outlines a generic VFA algorithm, and briefly discusses advanced topics of VFA. Second, the VFA algorithm is applied to the taxicab problem that describes an easy-to-understand transportation planning task. Detailed step-by-step results are presented for a small-scale instance, allowing readers to gain an intuition about VFA’s main principles. Third, larger instances are solved by enhancing the basic VFA algorithm demonstrating its general capability to approach more complex problems. The experiments are done with artificial instances and the respective Python scripts are part of an electronic appendix. Overall, the tutorial provides the necessary knowledge to apply VFA to a wide range of stochastic and dynamic settings and addresses likewise researchers, lecturers, tutors, students, and practitioners.

Appendices

Appendix A. Python scripts

The Python scripts with the implementations of the two solution approaches (backward dynamic programming, VFA) are available as additional files in the electronic appendix of this paper and in the following repository: https://github.com/aheiX/Tutorial-on-Value-Function-Approximation. These scripts can be used to reproduce results from the experimental study. All data and material is also available from the corresponding author on request.

Appendix B. Additional algorithms

This section describes two algorithms: first, Algorithm 2 provides a generic description of how to solve the Bellman equation backwardly and dynamically and, second, Algorithm 3 extends the general VFA algorithm (see Algorithm 1 in Section 5) with a greedy exploration strategy.

Appendix C. Optimal policy in a small-scale example

This section provides details on the optimal policy of the small-scale example used in Section 5. Figure 10  shows the decision tree. Notice that the decision tree consists of repetitive sub-problems. For example, being in $t = 2$ in node A appears three times in the decision tree: (i) if $x_{0} = \text {A}$ and $x_{1} = \text {A}$, (ii) if $x_{0} = \text {B}$ and $x_{1} = \text {A}$, and (iii) if $x_{0} = \text {C}$ and $x_{1} = \text {A}$. In backward dynamic programming, values of sub-problems are memoized once they are calculated for the first time, which reduces the total computation time. In the decision tree, values atop of the nodes correspond to expected values following an optimal policy. Such policy considers the immediate reward (resulting from the demand) as well as the downstream value (resulting from future periods). These values are calculated backwardly and dynamically, i.e., first values in $t = 2$ are calculated, then values in $t = 1$, and so on. As explained in Section 4.3, the value of the optimal policy does not simply result from choosing the move with the highest expected value. Instead, it is calculated by considering the optimal decision under each possible demand-combination. The algorithm for this is provided in Fig. 4b and exemplary applied to nodes A, B, and A of periods 2, 1, and 0 in Formulas (20)–(22), respectively.

$$\begin{aligned} V_{2}(S_{2}|l_{2}=A) =~&(0.33) \cdot 66~ \nonumber \\&+((1-0.33) \cdot 0.61) \cdot 46~ \nonumber \\&+((1 - 0.33 - (1-0.33) \cdot 0.61) \cdot 0.36) \cdot 18 = 42.27 \end{aligned}$$

(20)

$$\begin{aligned} V_{1}(S_{1}|l_{1}=B) =~&\underbrace{(0.60 )}_{p^{r}_{0}}\cdot (72 + \gamma \cdot 79.61)~\nonumber \\&+\underbrace{((1-p^{r}_{0}) \cdot 0.79)}_{p^{r}_{1}} \cdot (33 + \gamma \cdot 42.27)~\nonumber \\&+\underbrace{((1 - p^{r}_{0} - p^{r}_{1}) \cdot 0.39)}_{p^{r}_{2}} \cdot (13 + \gamma \cdot 66.27)~\nonumber \\&+(1 - p^{r}_{0} - p^{r}_{1} - p^{r}_{2}) \cdot (0 + \gamma \cdot 79.61) = 107.96 \end{aligned}$$

(21)

$$\begin{aligned} V_{0}(S_{0}|l_{0}=A) =~&\underbrace{(0.49 )}_{p^{r}_{0}} \cdot (98 + \gamma \cdot 92.01)~\nonumber \\&+\underbrace{((1-p^{r}_{0}) \cdot 0.62)}_{p^{r}_{1}} \cdot (52 + \gamma \cdot 107.96)~\nonumber \\&+\underbrace{((1 - p^{r}_{0} - p^{r}_{1}) \cdot 0.74)}_{p^{r}_{2}} \cdot (28 + \gamma \cdot 115.15)~\nonumber \\&+(1 - p^{r}_{0} - p^{r}_{1} - p^{r}_{2}) \cdot (0 + \gamma \cdot 115.15) = 149.71 \end{aligned}$$

(22)

Fig. 10

Decision tree for the example problem. Values atop of the arcs are the demand and its probability and the values atop of the decision nodes are the discounted downstream value following an optimal policy