Surgery scheduling of pelvic fracture patients with stochastic recovery time

Abstract

Pelvic fracture is a severe trauma and is often seen in the traffic accidents, which are associated with complications or multiple injuries. Surgery is the main treatment for patients with serious conditions, while conservative treatment is adopted for older or minor-illness patients. Surgery resources, such as doctors, nurses, and operating rooms, are shared by all pelvic fracture patients. From the perspective of patient state, this paper divides patients who require surgery into two types, convalescent patients and scheduled patients. Convalescent patients’ life states are always unstable, and they require recovery time to meet the condition of surgery. The recovery time is usually stochastic due to different patient situations. Scheduled patients have stable life states, and the pelvic fracture surgical plan is scheduled days or weeks in advance. Considering the characteristics of the two types of patients, a finite-horizon Markov decision process (MDP) model is established. With data collected from the hospital, parameters are set and experiments are designed to reveal the dynamic priority rules for receiving patients into surgery. Performances of different scenarios are compared, and the optimal policies obtained from the MDP are analyzed.

Appendices

Appendix: Proofs

Lemma A1:

For each profit functions $g \in G_{t + 1}$, $A_{t}^{\alpha } \in G_{t} ,t = 1, \ldots ,T$.

Lemma A2

For each service period $t = 1,2, \ldots ,T + 1$, and each state $\left( {c_{{t,rt_{0} }} ,s_{t} } \right) \in S_{t - 1}$, if there is $r_{c} \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right)$, $r_{s} \ge g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right)$, then the following two formulas is satisfied.

Proof of Proposition 1

The optimal reward function is established as the following equation:

$$ \begin{aligned} & V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right) = - c_{{t,rt_{0} }} \cdot w_{c} - s_{t} \cdot w_{s} \\&\quad\quad+ \left[ \begin{gathered} p_{{t,rt_{1} }} \cdot \left( {1 - p_{s} \cdot a_{t + 1} } \right) \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1 + c_{{t,rt_{1} }} ,s_{t} } \right) \hfill \\ + p_{{t,rt_{1} }} \cdot p_{s} \cdot a_{t + 1} \cdot p_{{t,rt_{2} }} \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1 + c_{{t,rt_{1} }} ,s_{t} + 1} \right) \hfill \\ + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left( {1 - p_{s} \cdot a_{t + 1} } \right) \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + c_{{t,rt_{1} }} ,s_{t} } \right) \hfill \\ + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot p_{s} \cdot a_{t + 1} \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + c_{{t,rt_{1} }} ,s_{t} + 1} \right) \hfill \\ \end{gathered} \right] \\ & \quad = - c_{{t,rt_{0} }} \cdot w_{c} - s_{t} \cdot w_{s} + p_{{t,rt_{1} }} \cdot \left( {1 - p_{s} \cdot a_{t + 1} } \right) \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1 + c_{{t,rt_{1} }} ,s_{t} } \right) \\ & \quad \quad + p_{{t,rt_{1} }} \cdot p_{s} \cdot a_{t + 1} \cdot p_{{t,rt_{2} }} \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1 + c_{{t,rt_{1} }} ,s_{t} + 1} \right) \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left( {1 - p_{s} \cdot a_{t + 1} } \right) \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + c_{{t,rt_{1} }} ,s_{t} } \right) \\ & \quad \quad+ \left( {1 - p_{{t,rt_{1} }} } \right) \cdot p_{s} \cdot a_{t + 1} \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + c_{{t,rt_{1} }} ,s_{t} + 1} \right) \\ \end{aligned} $$

(11)

For $t = 1,2, \ldots ,T + 1$, $G_{t}$ is defined as the class of profit functions on state $S_{t}$. So for each $g \in G_{t}$, which evaluates the states belong to $S_{t}$, we have got the following formulas.

$$ g\left( {c_{{rt_{0} }} ,s + 1} \right) - g\left( {c_{{rt_{0} }} + 1,s} \right) \le g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - g\left( {c_{{rt_{0} }} + 2,s} \right) $$

(12)

$$ g\left( {c_{{rt_{0} }} + 1,s} \right) - g\left( {c_{{rt_{0} }} ,s + 1} \right) \le g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - g\left( {c_{{rt_{0} }} ,s + 2} \right) $$

(13)

$$ g\left( {c_{{rt_{0} }} ,s} \right) - g\left( {c_{{rt_{0} }} ,s + 1} \right) \le g\left( {c_{{rt_{0} }} + 1,s} \right) - g\left( {c_{{rt_{0} }} + 1,s + 1} \right) $$

(14)

$$ g\left( {c_{{rt_{0} }} ,s} \right) - g\left( {c_{{rt_{0} }} + 1,s} \right) \le g\left( {c_{{rt_{0} }} + 1,s} \right) - g\left( {c_{{rt_{0} }} + 2,s} \right) $$

(15)

$$ g\left( {c_{{rt_{0} }} ,s} \right) - g\left( {c_{{rt_{0} }} ,s + 1} \right) \le g\left( {c_{{rt_{0} }} ,s + 1} \right) - g\left( {c_{{rt_{0} }} ,s + 2} \right) $$

(16)

It will be shown that the class $G_{t + 1}$ is mapped onto the class $G_{t}$ by the action $A_{t}^{\alpha }$:

Lemma A1:

For each profit functions $g \in G_{t + 1}$, $A_{t}^{\alpha } \in G_{t} ,t = 1, \ldots ,T$.

Lemma A2:

For each service period $t = 1,2, \ldots ,T + 1$, and each state $\left( {c_{{t,rt_{0} }} ,s_{t} } \right) \in S_{t - 1}$, if there is $r_{c} \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right)$, $r_{s} \ge g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right)$, then the following two formulas is satisfied.

Proof of Lemma A1

Equation (2) can be written as:$g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - g\left( {c_{{rt_{0} }} + 2,s} \right) - g\left( {c_{{rt_{0} }} ,s + 1} \right) + g\left( {c_{{rt_{0} }} + 1,s} \right) \ge 0$.

In the aspect of the following formula: $A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 2,s} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right)$

4 different conditions are considered respectively, and $a_{t + 1}$ is assumed to be 0.

(1) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} \ge 1$

$$ \begin{aligned} & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 2,s} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 3,s_{t} } \right)}\right.\\ & \quad\left.{ - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) }\right.\\ & \quad\left.{- H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right)} \right] \\ \end{aligned} $$

In this condition, there are two expressions, the first one is explained detailed as the following, while the second one share the similar structure.

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 3,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 3,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ \end{aligned} $$

1) If $g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s}$, and $g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 3,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 3,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - r_{c} - g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - r_{c} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) \ge 0 \\ \end{aligned} $$

2) If $g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s}$, and $g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 3,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) &- H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 3,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} + 3,s_{t} - 1} \right) - r_{s} - g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - r_{s} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) \\ & \quad \quad- g\left( {c_{{t,rt_{0} }} + 3,s_{t} - 1} \right) - g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \ge 0 \\ \end{aligned} $$

3) If $g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s}$, and $g\left( {c_{{rt_{0} }} + 2,s} \right) + r_{c} \le g\left( {c_{{rt_{0} }} + 3,s - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 3,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - r_{c} - g\left( {c_{{t,rt_{0} }} + 3,s_{t} - 1} \right) - r_{s} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) - g\left( {c_{{t,rt_{0} }} + 3,s_{t} - 1} \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} + 2,s} \right)s_{t} - g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) \ge 0 \\ \end{aligned} $$

4) If $g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s}$, and $g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 3,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 3,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - r_{s} - g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - r_{c} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right){ + }g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) = 0 \\ \end{aligned} $$

(2) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} = 0$

$$ \begin{aligned} & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 2,s} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) \\ & \quad { = }A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 2,0} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,1} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,0} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 3,0} \right) }\right. \\ & \quad \quad\left.{- H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) }\right. \\ & \quad\quad \left.{ - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right)} \right] \\ \end{aligned} $$

There are two expressions in the above formula, the first one is explained detailed as the following, while the second one share the similar structure.

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 3,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,0} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {c_{{t,rt_{0} }} ,1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} + 2,0} \right) + g\left( {c_{{t,rt_{0} }} + 1,0} \right) \\ \end{aligned} $$

1) If $g\left( {c_{{t,rt_{0} }} ,1} \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 1,0} \right) + r_{s}$

$$ \begin{gathered} H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 3,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) \hfill \\ \ge g\left( {c_{{t,rt_{0} }} + 1,1} \right) - g\left( {c_{{t,rt_{0} }} ,1} \right) - g\left( {c_{{t,rt_{0} }} + 2,0} \right) + g\left( {c_{{t,rt_{0} }} + 1,0} \right) \ge 0 \hfill \\ \end{gathered} $$

2) If $g\left( {c_{{t,rt_{0} }} ,1} \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 1,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 3,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 2,0} \right) - g\left( {c_{{t,rt_{0} }} + 1,0} \right) - g\left( {c_{{t,rt_{0} }} + 2,0} \right) + g\left( {c_{{t,rt_{0} }} + 1,0} \right) = 0 \\ \end{aligned} $$

(3) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} \ge 1$

$$ \begin{aligned} & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 2,s} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) \\ & \quad { = }A_{t}^{\alpha } g\left( {1,s + 1} \right) - A_{t}^{\alpha } g\left( {2,s} \right) - A_{t}^{\alpha } g\left( {0,s + 1} \right) + A_{t}^{\alpha } g\left( {1,s} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {3,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {2,s_{t} } \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} } \right)} \right] \\ \end{aligned} $$

There are two expressions in the above formula, the second one is explained detailed as the following:

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) \\ & \quad = \max \left[ {g\left( {0,s_{t} + 1} \right) + r_{c} ,g\left( {1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {1,s_{t} } \right) + r_{c} ,g\left( {2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {0,s_{t} } \right) + r_{c} ,g\left( {1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {0,s_{t} } \right) - r_{s} \\ \end{aligned} $$

1) If $g\left( {1,s_{t} } \right) + r_{c} \ge g\left( {2,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) \\ & \quad = \max \left[ {g\left( {0,s_{t} + 1} \right) + r_{c} ,g\left( {1,s_{t} } \right) + r_{s} } \right] - g\left( {1,s_{t} } \right) - r_{c} \\ & \quad \quad + \max \left[ {g\left( {0,s_{t} } \right) + r_{c} ,g\left( {1,s_{t} - 1} \right) + r_{s} } \right] - g\left( {0,s_{t} } \right) - r_{s} \\ & \quad \ge g\left( {1,s_{t} } \right) + r_{s} - g\left( {1,s_{t} } \right) - r_{c} + g\left( {0,s_{t} } \right) + r_{c} - g\left( {0,s_{t} } \right) - r_{s} = 0 \\ \end{aligned} $$

2) If $g\left( {1,s_{t} } \right) + r_{c} \le g\left( {2,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) \\ & \quad = \max \left[ {g\left( {0,s_{t} + 1} \right) + r_{c} ,g\left( {1,s_{t} } \right) + r_{s} } \right] - g\left( {2,s_{t} - 1} \right) - r_{s} \\ & \quad \quad + \max \left[ {g\left( {0,s_{t} } \right) + r_{c} ,g\left( {1,s_{t} - 1} \right) + r_{s} } \right] - g\left( {0,s_{t} } \right) - r_{s} \\ & \quad \ge g\left( {1,s_{t} } \right) + r_{s} - g\left( {2,s_{t} - 1} \right) - r_{s} + g\left( {1,s_{t} - 1} \right) + r_{s} - g\left( {0,s_{t} } \right) - r_{s} \ge 0 \\ \end{aligned} $$

The first expression’s prove is same as the case of ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} \ge 1$.

(4) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} = 0$

$$ \begin{aligned} & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 2,s} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) \\ & \quad { = }A_{t}^{\alpha } g\left( {1,1} \right) - A_{t}^{\alpha } g\left( {2,0} \right) - A_{t}^{\alpha } g\left( {0,1} \right) + A_{t}^{\alpha } g\left( {1,0} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {2,1} \right) - H_{t + 1}^{\alpha } \left( {3,0} \right) - H_{t + 1}^{\alpha } \left( {1,1} \right) + H_{t + 1}^{\alpha } \left( {2,0} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {1,1} \right) - H_{t + 1}^{\alpha } \left( {2,0} \right) - H_{t + 1}^{\alpha } \left( {0,1} \right) + H_{t + 1}^{\alpha } \left( {1,0} \right)} \right] \\ \end{aligned} $$

There are two expressions in the above formula, the first one is explained detailed as the following:

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {2,1} \right) - H_{t + 1}^{\alpha } \left( {3,0} \right) - H_{t + 1}^{\alpha } \left( {1,1} \right) + H_{t + 1}^{\alpha } \left( {2,0} \right) \\ & \quad = \max \left[ {g\left( {1,1} \right) + r_{c} ,g\left( {2,0} \right) + r_{s} } \right] - \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {2,0} \right) - r_{c} + g\left( {1,0} \right) + r_{c} \\ \end{aligned} $$

1) If $g\left( {0,1} \right) + r_{c} \ge g\left( {1,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {2,1} \right) - H_{t + 1}^{\alpha } \left( {3,0} \right) - H_{t + 1}^{\alpha } \left( {1,1} \right) + H_{t + 1}^{\alpha } \left( {2,0} \right) \\ & \quad = \max \left[ {g\left( {1,1} \right) + r_{c} ,g\left( {2,0} \right) + r_{s} } \right] - \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {2,0} \right) + g\left( {1,0} \right) \\ & \quad \ge g\left( {1,1} \right) + r_{c} - g\left( {0,1} \right) - r_{c} - g\left( {2,0} \right) + g\left( {1,0} \right) \ge 0 \\ \end{aligned} $$

2) If $g\left( {0,1} \right) + r_{c} \le g\left( {1,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {2,1} \right) - H_{t + 1}^{\alpha } \left( {3,0} \right) - H_{t + 1}^{\alpha } \left( {1,1} \right) + H_{t + 1}^{\alpha } \left( {2,0} \right) \\ & \quad = \max \left[ {g\left( {1,1} \right) + r_{c} ,g\left( {2,0} \right) + r_{s} } \right] - \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {2,0} \right) + g\left( {1,0} \right) \\ & \quad \ge g\left( {2,0} \right) + r_{s} - g\left( {1,0} \right) - r_{s} - g\left( {2,0} \right) + g\left( {1,0} \right) = 0 \\ \end{aligned} $$

The second one is explained detailed as the following:

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,1} \right) - H_{t + 1}^{\alpha } \left( {2,0} \right) - H_{t + 1}^{\alpha } \left( {0,1} \right) + H_{t + 1}^{\alpha } \left( {1,0} \right) \\ & \quad = \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {1,0} \right) - r_{c} - g\left( {0,0} \right) - r_{s} + g\left( {0,0} \right) + r_{c} \\ & \quad = \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] - g\left( {1,0} \right) - r_{s} \\ \end{aligned} $$

1) If $g\left( {0,1} \right) + r_{c} \ge g\left( {1,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,1} \right) - H_{t + 1}^{\alpha } \left( {2,0} \right) - H_{t + 1}^{\alpha } \left( {0,1} \right) + H_{t + 1}^{\alpha } \left( {1,0} \right) \\ & \quad = g\left( {0,1} \right) + r_{c} - g\left( {1,0} \right) - r_{s} \ge 0 \\ \end{aligned} $$

2) If $g\left( {0,1} \right) + r_{c} \le g\left( {1,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,1} \right) - H_{t + 1}^{\alpha } \left( {2,0} \right) - H_{t + 1}^{\alpha } \left( {0,1} \right) + H_{t + 1}^{\alpha } \left( {1,0} \right) \\ & \quad = g\left( {1,0} \right) + r_{s} - g\left( {1,0} \right) - r_{s} = 0 \\ \end{aligned} $$

From above all, it can be seen that:$A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 2,s} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) \ge 0$.

So, Eq. (2) $g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - g\left( {c_{{rt_{0} }} + 2,s} \right) - g\left( {c_{{rt_{0} }} ,s + 1} \right) + g\left( {c_{{rt_{0} }} + 1,s} \right) \ge 0$ is satisfied. The proof of Eq. (3) is same as the proof of Eq. (2).

Equation (3) can be written as:$g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - g\left( {c_{{rt_{0} }} ,s + 2} \right) - g\left( {c_{{rt_{0} }} + 1,s} \right) + g\left( {c_{{rt_{0} }} ,s + 1} \right) \ge 0$.

In the aspect of the following formula: $A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 2} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right)$

4 different conditions are considered respectively, and $a_{t + 1}$ is assumed to be 0.

(1) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} \ge 1$

$$ \begin{aligned} & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 2} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 2} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) }\right. \\ & \quad \quad\left.{+ H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 2} \right) }\right. \\ & \quad\quad \left.{- H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right)} \right] \\ \end{aligned} $$

In this condition, there are two expressions in the above formula, the second one is explained detailed as the following, while the first one share the similar structure.

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 2} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 2} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s} } \right] \\ \end{aligned} $$

1) If $g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s}$, and $g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 2} \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 2} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 2} \right) - r_{c} - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - r_{c} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 1} \right) - g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 2} \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) \ge 0 \\ \end{aligned} $$

2) If $g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s}$, and $g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 2} \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 2} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - r_{s} - g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) - r_{s} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) \ge 0 \\ \end{aligned} $$

3) If $g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s}$, and $g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 2} \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 2} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 2} \right) - r_{c} - g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) - r_{s} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 1} \right) + g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 2} \right) - g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) \\ & \quad \ge g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right)g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \ge 0 \\ \end{aligned} $$

4) If $g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s}$, and $g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 2} \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 2} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} - 1,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - r_{c} - g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - r_{s} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} + g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{s} - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - r_{c} - g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - r_{s} = 0 \\ \end{aligned} $$

(2) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} = 0$

$$ \begin{aligned} & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 2} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,2} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) }\right. \\ &\quad \quad \left.{+ H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) }\right. \\ &\quad \quad \left.{ - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,2} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right)} \right] \\ \end{aligned} $$

In this condition, there are two expressions in the above formula, the first one is explained detailed as the following:

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,2} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,0} \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} ,1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,0} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {c_{{t,rt_{0} }} ,2} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,1} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} + 1,0} \right) - r_{c} \\ \end{aligned} $$

1) If $g\left( {c_{{t,rt_{0} }} ,2} \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 1,1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,2} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,0} \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} ,1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} ,2} \right) - r_{c} - g\left( {c_{{t,rt_{0} }} + 1,0} \right) - r_{c} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 1,1} \right) + r_{c} + g\left( {c_{{t,rt_{0} }} ,1} \right) + r_{c} - g\left( {c_{{t,rt_{0} }} ,2} \right) - r_{c} - g\left( {c_{{t,rt_{0} }} + 1,0} \right) - r_{c} \ge 0 \\ \end{aligned} $$

2) If $g\left( {c_{{t,rt_{0} }} ,2} \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 1,1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,2} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,0} \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} ,1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} + 1,1} \right) - r_{s} - g\left( {c_{{t,rt_{0} }} + 1,0} \right) - r_{c} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 1,1} \right) + g\left( {c_{{t,rt_{0} }} + 1,0} \right) - g\left( {c_{{t,rt_{0} }} + 1,1} \right) - g\left( {c_{{t,rt_{0} }} + 1,0} \right) = 0 \\ \end{aligned} $$

(3) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} \ge 1$

$$ \begin{aligned} & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 2} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} + 2} \right) - H_{t + 1}^{\alpha } \left( {2,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 2} \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right)} \right] \\ \end{aligned} $$

In this condition, there are two expressions in the above formula, the second one is explained detailed as the following:

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 2} \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {0,s_{t} + 1} \right) + r_{c} ,g\left( {1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {0,s_{t} } \right) + r_{c} ,g\left( {1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {0,s_{t} + 1} \right) + g\left( {0,s_{t} } \right) \\ \end{aligned} $$

1) If $g\left( {0,s_{t} } \right) + r_{c} \ge g\left( {1,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 2} \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {0,s_{t} + 1} \right) + r_{c} ,g\left( {1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - g\left( {0,s_{t} } \right) - r_{c} - g\left( {0,s_{t} + 1} \right) + g\left( {0,s_{t} } \right) \\ & \quad = \max \left[ {g\left( {0,s_{t} + 1} \right) + r_{c} ,g\left( {1,s_{t} } \right) + r_{s} } \right] - g\left( {0,s_{t} + 1} \right) - r_{c} \\ & \quad \ge g\left( {0,s_{t} + 1} \right) + r_{c} - g\left( {0,s_{t} + 1} \right) - r_{c} = 0 \\ \end{aligned} $$

2) If $g\left( {0,s_{t} } \right) + r_{c} \le g\left( {1,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 2} \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {0,s_{t} + 1} \right) + r_{c} ,g\left( {1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - g\left( {1,s_{t} - 1} \right) - r_{s} - g\left( {0,s_{t} + 1} \right) + g\left( {0,s_{t} } \right) \\ & \quad \ge g\left( {1,s_{t} } \right) + r_{s} - g\left( {1,s_{t} - 1} \right) - r_{s} - g\left( {0,s_{t} + 1} \right) + g\left( {0,s_{t} } \right) \ge 0 \\ \end{aligned} $$

(4) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} = 0$

$$ \begin{aligned} & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 2} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {2,1} \right) - H_{t + 1}^{\alpha } \left( {1,2} \right) - H_{t + 1}^{\alpha } \left( {2,0} \right) + H_{t + 1}^{\alpha } \left( {1,1} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {1,1} \right) - H_{t + 1}^{\alpha } \left( {0,2} \right) - H_{t + 1}^{\alpha } \left( {1,0} \right) + H_{t + 1}^{\alpha } \left( {0,1} \right)} \right] \\ \end{aligned} $$

In this condition, there are two expressions in the above formula, the first one is explained detailed as the following:

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {2,1} \right) - H_{t + 1}^{\alpha } \left( {1,2} \right) - H_{t + 1}^{\alpha } \left( {2,0} \right) + H_{t + 1}^{\alpha } \left( {1,1} \right) \\ & \quad = \max \left[ {g\left( {1,1} \right) + r_{c} ,g\left( {2,0} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {0,2} \right) + r_{c} ,g\left( {1,1} \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] - g\left( {1,0} \right) - r_{c} \\ \end{aligned} $$

1) If $g\left( {0,2} \right) + r_{c} \ge g\left( {1,1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {2,1} \right) - H_{t + 1}^{\alpha } \left( {1,2} \right) - H_{t + 1}^{\alpha } \left( {2,0} \right) + H_{t + 1}^{\alpha } \left( {1,1} \right) \\ & \quad = \max \left[ {g\left( {1,1} \right) + r_{c} ,g\left( {2,0} \right) + r_{s} } \right] + \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {0,2} \right) - r_{c} - g\left( {1,0} \right) - r_{c} \\ & \quad \ge g\left( {1,1} \right) + r_{c} + g\left( {0,1} \right) + r_{c} - g\left( {0,2} \right) - r_{c} - g\left( {1,0} \right) - r_{c} \ge 0 \\ \end{aligned} $$

2) If $g\left( {0,2} \right) + r_{c} \le g\left( {1,1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {2,1} \right) - H_{t + 1}^{\alpha } \left( {1,2} \right) - H_{t + 1}^{\alpha } \left( {2,0} \right) + H_{t + 1}^{\alpha } \left( {1,1} \right) \\ & \quad = \max \left[ {g\left( {1,1} \right) + r_{c} ,g\left( {2,0} \right) + r_{s} } \right] + \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {1,1} \right) - r_{s} - g\left( {1,0} \right) - r_{c} \\ & \quad \ge g\left( {1,1} \right) + r_{c} + g\left( {1,0} \right) + r_{s} - g\left( {1,1} \right) - r_{s} - g\left( {1,0} \right) - r_{c} = 0 \\ \end{aligned} $$

For the second expression:

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,1} \right) - H_{t + 1}^{\alpha } \left( {0,2} \right) - H_{t + 1}^{\alpha } \left( {1,0} \right) + H_{t + 1}^{\alpha } \left( {0,1} \right) \\ & \quad \quad \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {0,1} \right) - r_{s} - g\left( {0,0} \right) - r_{c} + g\left( {0,0} \right) + r_{s} \\ & \quad = \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] - g\left( {0,1} \right) - r_{c} \\ & \quad \ge g\left( {0,1} \right) + r_{c} - g\left( {0,1} \right) - r_{c} = 0 \\ \end{aligned} $$

Then we introduce another Lemma to prove Eq. (4).

Lemma A2

For each service period $t = 1,2, \ldots ,T + 1$, and each state $\left( {c_{{t,rt_{0} }} ,s_{t} } \right) \in S_{t - 1}$, if there is $r_{c} \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right)$, $r_{s} \ge g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right)$, then the following two formulas is satisfied.

$$ r_{c} \ge A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) $$

$$ r_{s} \ge A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) $$

Proof of Lemma A2

First, we consider $r_{c} \ge A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} ,s_{t} } \right)$, and $a_{t + 1}$ is assumed to be 0. 4 different conditions are considered respectively.

$$ \begin{aligned} & r_{c} + A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \\ & \quad = r_{c} + w_{c} + p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right)} \right] \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {\left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {\left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right)} \right] \\ \end{aligned} $$

(1) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} \ge 1$.

In this condition, there are two expressions in the above equation, the first one is explained detailed as the following, while the second one share the similar structure.

$$ \begin{aligned} & \left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = r_{c} + w_{c} + \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ \end{aligned} $$

1) If $g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & \left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = r_{c} + w_{c} - g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - r_{c} \\ & \quad \quad \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \ge r_{c} + w_{c} { + }g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} - g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - r_{c} \\ & \quad { = }r_{c} + w_{c} { + }g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) > 0 \\ \end{aligned} $$

2) If $g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & \left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = r_{c} + w_{c} - g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) - r_{s} \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \ge r_{c} + w_{c} + g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s} - g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) - r_{s} \\ & \quad = r_{c} + w_{c} + g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) - g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) > 0 \\ \end{aligned} $$

(2) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} = 0$

$$ \begin{aligned} & r_{c} + A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \\ & \quad = r_{c} + A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} ,0} \right) - A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} + 1,0} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {\left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {\left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right)} \right] \\ \end{aligned} $$

There are two expressions in the above equation, the first one is explained detailed as the following, while the second one share the similar structure.

$$ \begin{aligned} & \left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) \\ & \quad = r_{c} + w_{c} + g\left( {c_{{t,rt_{0} }} ,0} \right) + r_{c} - g\left( {c_{{t,rt_{0} }} + 1,0} \right) - r_{c} \\ & \quad = r_{c} + w_{c} + g\left( {c_{{t,rt_{0} }} ,0} \right) - g\left( {c_{{t,rt_{0} }} + 1,0} \right) > 0 \\ \end{aligned} $$

(3) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} \ge 1$

$$ \begin{aligned} & r_{c} + A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \\ & \quad = r_{c} + A_{t}^{\alpha } g\left( {0,s_{t} } \right) - A_{t}^{\alpha } g\left( {1,s_{t} } \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {\left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {2,s_{t} } \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {\left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {0,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} } \right)} \right] \\ \end{aligned} $$

The proof of the first expression is same as the case of ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} \ge 1$. So, the second expression is considered.

$$ \begin{aligned} & \left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {0,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) \\ & \quad = r_{c} + w_{c} + g\left( {0,s_{t} - 1} \right) + r_{s} \\ & \quad \quad - \max \left[ {g\left( {0,s_{t} } \right) + r_{c} ,g\left( {1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad = \left\{ {\begin{array}{*{20}c} {w_{c} + r_{s} + g\left( {0,s_{t} - 1} \right) - g\left( {0,s_{t} } \right) > 0,{\kern 1pt} {\kern 1pt} {\kern 1pt} if{\kern 1pt} {\kern 1pt} {\kern 1pt} g\left( {0,s_{t} } \right) + r_{c} \ge g\left( {1,s_{t} - 1} \right) + r_{s} } \\ {r_{c} + w_{c} + g\left( {0,s_{t} - 1} \right) - g\left( {1,s_{t} - 1} \right) > 0,{\kern 1pt} {\kern 1pt} {\kern 1pt} if{\kern 1pt} {\kern 1pt} {\kern 1pt} g\left( {0,s_{t} } \right) + r_{c} \ge g\left( {1,s_{t} - 1} \right) + r_{s} } \\ \end{array} } \right. \\ \end{aligned} $$

(4) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} = 0$

$$ \begin{aligned} & r_{c} + A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \\ & \quad = r_{c} + A_{t}^{\alpha } g\left( {0,0} \right) - A_{t}^{\alpha } g\left( {1,0} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {\left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {1,0} \right) - H_{t + 1}^{\alpha } \left( {2,0} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {\left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {0,0} \right) - H_{t + 1}^{\alpha } \left( {1,0} \right)} \right] \\ \end{aligned} $$

The proof of the first expression is same as the case of ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} = 0$. So, the second expression is considered.

$$ \begin{aligned} & \left( {r_{c} + w_{c} } \right) + H_{t + 1}^{\alpha } \left( {0,0} \right) - H_{t + 1}^{\alpha } \left( {1,0} \right) \\ & \quad = r_{c} + w_{c} + g\left( {0,0} \right) - g\left( {0,0} \right) - r_{c} = w_{c} > 0 \\ \end{aligned} $$

From the above, we can get the conclusion that $r_{c} + A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \ge 0$. $r_{s} + A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - A_{t}^{\alpha } g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) \ge 0$ can be validated in the similar way.

The proof of Eq. (4) is conducted the same way as above.

Equation (4) can be written as:$g\left( {c_{{rt_{0} }} + 1,s} \right) - g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - g\left( {c_{{rt_{0} }} ,s} \right) + g\left( {c_{{rt_{0} }} ,s + 1} \right) \ge 0$.

In the aspect of the following formula: $A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right)$

4 different conditions are considered respectively, and $a_{t + 1}$ is assumed to be 0.

(1) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} \ge 1$

$$ \begin{aligned} & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) }\right. \\ &\quad \quad \left.{- H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) }\right. \\ &\quad \quad \left.{- H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right)} \right] \\ \end{aligned} $$

In this condition, there are two expressions in the above formula, the first one is explained detailed as the following, while the second one share the similar structure.

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ \end{aligned} $$

1) If $g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{s}$, and $g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - r_{c} - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - r_{c} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) \ge 0 \\ \end{aligned} $$

2) If $g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{s}$, and $g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad - g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - r_{s} - g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) \ge 0 \\ \end{aligned} $$

3) If $g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{s}$, and $g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - r_{c} - g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right){ + }g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) \ge 0 \\ \end{aligned} $$

4) If $g\left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) + r_{s}$, and $g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {g\left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - r_{s} - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) - r_{c} \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + g\left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - g\left( {c_{{t,rt_{0} }} ,s_{t} } \right) \ge 0 \\ \end{aligned} $$

(2) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} = 0$

$$ \begin{aligned} & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) \\ & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,0} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,0} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,1} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right)} \right] \\ \end{aligned} $$

In this condition, there are two expressions in the above formula, the first one is explained detailed as the following, while the second one share the similar structure.

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} ,1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,0} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {c_{{t,rt_{0} }} + 1,1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 2,0} \right) + r_{s} } \right] \\ & \quad \quad + g\left( {c_{{t,rt_{0} }} + 1,0} \right) + r_{c} - g\left( {c_{{t,rt_{0} }} ,0} \right) - r_{c} \\ \end{aligned} $$

1) If $g\left( {c_{{t,rt_{0} }} + 1,1} \right) + r_{c} \ge g\left( {c_{{t,rt_{0} }} + 2,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} ,1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} + 1,1} \right) - r_{c} + g\left( {c_{{t,rt_{0} }} + 1,0} \right) - g\left( {c_{{t,rt_{0} }} ,0} \right) \\ & \quad \ge g\left( {c_{{t,rt_{0} }} ,1} \right) - g\left( {c_{{t,rt_{0} }} + 1,1} \right) + g\left( {c_{{t,rt_{0} }} + 1,0} \right) - g\left( {c_{{t,rt_{0} }} ,0} \right) \ge 0 \\ \end{aligned} $$

2) If $g\left( {c_{{t,rt_{0} }} + 1,1} \right) + r_{c} \le g\left( {c_{{t,rt_{0} }} + 2,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) \\ & \quad = \max \left[ {g\left( {c_{{t,rt_{0} }} ,1} \right) + r_{c} ,g\left( {c_{{t,rt_{0} }} + 1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {c_{{t,rt_{0} }} + 2,0} \right) - r_{s} + g\left( {c_{{t,rt_{0} }} + 1,0} \right) - g\left( {c_{{t,rt_{0} }} ,0} \right) \\ & \quad \ge g\left( {c_{{t,rt_{0} }} + 1,0} \right) + g\left( {c_{{t,rt_{0} }} + 1,0} \right) - g\left( {c_{{t,rt_{0} }} + 2,0} \right) - g\left( {c_{{t,rt_{0} }} ,0} \right) \ge 0 \\ \end{aligned} $$

(3) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} \ge 1$

$$ \begin{aligned} & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) \\ & \quad = A_{t}^{\alpha } g\left( {1,s} \right) - A_{t}^{\alpha } g\left( {1,s + 1} \right) - A_{t}^{\alpha } g\left( {0,s} \right) + A_{t}^{\alpha } g\left( {0,s + 1} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {2,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right)} \right] \\ \end{aligned} $$

The proof of the first expression is same as the case of ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} \ge 1$. So, the second expression is considered.

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {0,s_{t} } \right) + r_{c} ,g\left( {1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {0,s_{t} + 1} \right) + r_{c} ,g\left( {1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - g\left( {0,s_{t} - 1} \right) + g\left( {0,s_{t} } \right) \\ \end{aligned} $$

1) If $g\left( {0,s_{t} + 1} \right) + r_{c} \ge g\left( {1,s_{t} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {0,s_{t} } \right) + r_{c} ,g\left( {1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {0,s_{t} + 1} \right) - r_{c} - g\left( {0,s_{t} - 1} \right) + g\left( {1,s_{t} } \right) \\ & \quad \ge g\left( {0,s_{t} } \right) - g\left( {0,s_{t} + 1} \right) - g\left( {0,s_{t} - 1} \right) + g\left( {0,s_{t} } \right) \ge 0 \\ \end{aligned} $$

2) If $g\left( {0,s_{t} + 1} \right) + r_{c} \le g\left( {1,s_{t} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} } \right) + H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) \\ & \quad = \max \left[ {g\left( {0,s_{t} } \right) + r_{c} ,g\left( {1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {1,s_{t} } \right) - r_{s} - g\left( {0,s_{t} - 1} \right) + g\left( {0,s_{t} } \right) \\ & \quad \ge g\left( {1,s_{t} - 1} \right) - g\left( {1,s_{t} } \right) - g\left( {0,s_{t} - 1} \right) + g\left( {0,s_{t} } \right) \ge 0 \\ \end{aligned} $$

(4) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} = 0$

$$ \begin{aligned} & A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s} \right) + A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s + 1} \right) \\ & \quad = A_{t}^{\alpha } g\left( {1,0} \right) - A_{t}^{\alpha } g\left( {1,1} \right) - A_{t}^{\alpha } g\left( {0,0} \right) + A_{t}^{\alpha } g\left( {0,1} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 1}^{\alpha } \left( {2,0} \right) - H_{t + 1}^{\alpha } \left( {2,1} \right) - H_{t + 1}^{\alpha } \left( {1,0} \right) + H_{t + 1}^{\alpha } \left( {1,1} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 1}^{\alpha } \left( {1,0} \right) - H_{t + 1}^{\alpha } \left( {1,1} \right) - H_{t + 1}^{\alpha } \left( {0,0} \right) + H_{t + 1}^{\alpha } \left( {0,1} \right)} \right] \\ \end{aligned} $$

There are two expressions in the above equation, the first one is explained detailed as the following,

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {2,0} \right) - H_{t + 1}^{\alpha } \left( {2,1} \right) - H_{t + 1}^{\alpha } \left( {1,0} \right) + H_{t + 1}^{\alpha } \left( {1,1} \right) \\ & \quad = \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {g\left( {1,1} \right) + r_{c} ,g\left( {2,0} \right) + r_{s} } \right] \\ & \quad \quad + g\left( {1,0} \right) + r_{c} - g\left( {0,0} \right) - r_{c} \\ \end{aligned} $$

1) If $g\left( {1,1} \right) + r_{c} \ge g\left( {2,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {2,0} \right) - H_{t + 1}^{\alpha } \left( {2,1} \right) - H_{t + 1}^{\alpha } \left( {1,0} \right) + H_{t + 1}^{\alpha } \left( {1,1} \right) \\ & \quad = \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {1,1} \right) - r_{c} + g\left( {1,0} \right) - g\left( {0,0} \right) \\ & \quad \ge g\left( {0,1} \right) - g\left( {1,1} \right) + g\left( {1,0} \right) - g\left( {0,0} \right) \ge 0 \\ \end{aligned} $$

2) If $g\left( {1,1} \right) + r_{c} \le g\left( {2,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {2,0} \right) - H_{t + 1}^{\alpha } \left( {2,1} \right) - H_{t + 1}^{\alpha } \left( {1,0} \right) + H_{t + 1}^{\alpha } \left( {1,1} \right) \\ & \quad = \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - g\left( {2,0} \right) - r_{s} + g\left( {1,0} \right) - g\left( {0,0} \right) \\ & \quad \ge g\left( {1,0} \right) - g\left( {2,0} \right) + g\left( {1,0} \right) - g\left( {0,0} \right) \ge 0 \\ \end{aligned} $$

For the second expression:

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,0} \right) - H_{t + 1}^{\alpha } \left( {1,1} \right) - H_{t + 1}^{\alpha } \left( {0,0} \right) + H_{t + 1}^{\alpha } \left( {0,1} \right) \\ & \quad = g\left( {0,0} \right) + r_{c} + g\left( {0,0} \right) + r_{s} - g\left( {0,0} \right) \\ & \quad \quad - \max \left[ {g\left( {0,1} \right) + r_{c} ,g\left( {1,0} \right) + r_{s} } \right] \\ \end{aligned} $$

1) If $g\left( {0,1} \right) + r_{c} \ge g\left( {1,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,0} \right) - H_{t + 1}^{\alpha } \left( {1,1} \right) - H_{t + 1}^{\alpha } \left( {0,0} \right) + H_{t + 1}^{\alpha } \left( {0,1} \right) \\ & \quad = g\left( {0,0} \right) + r_{c} + g\left( {0,0} \right) + r_{s} - g\left( {0,0} \right) - g\left( {0,1} \right) - r_{c} \\ & \quad = g\left( {0,0} \right) + r_{s} - g\left( {0,1} \right) \ge 0 \\ \end{aligned} $$

2) If $g\left( {0,1} \right) + r_{c} \le g\left( {1,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {1,0} \right) - H_{t + 1}^{\alpha } \left( {1,1} \right) - H_{t + 1}^{\alpha } \left( {0,0} \right) + H_{t + 1}^{\alpha } \left( {0,1} \right) \\ & \quad = g\left( {0,0} \right) + r_{c} + g\left( {0,0} \right) + r_{s} - g\left( {0,0} \right) - g\left( {1,0} \right) - r_{s} \\ & \quad = g\left( {0,0} \right) + r_{c} - g\left( {1,0} \right) \\ \end{aligned} $$

By the Lemma A2, we can see that $g\left( {0,0} \right) + r_{c} - g\left( {1,0} \right) \ge 0$.

By adding Eq. (2) and Eq. (4), we can get $g\left( {c_{{rt_{0} }} ,s} \right){ + }g\left( {c_{{rt_{0} }} + 2,s} \right) \le 2 \cdot g\left( {c_{{rt_{0} }} + 1,s} \right)$; by adding Eq. (3) and Eq. (4), we can get $g\left( {c_{{rt_{0} }} ,s} \right){ + }g\left( {c_{{rt_{0} }} ,s{ + }2} \right) \le 2 \cdot g\left( {c_{{rt_{0} }} ,s{ + }1} \right)$. So, we have the following two formulas.

$$ A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s} \right){ + }A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 2,s} \right) \le 2 \cdot A_{t}^{\alpha } g\left( {c_{{rt_{0} }} + 1,s} \right) $$

$$ A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s} \right){ + }A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s{ + }2} \right) \le 2 \cdot A_{t}^{\alpha } g\left( {c_{{rt_{0} }} ,s{ + }1} \right) $$

From the above, the proof of Lemma A1 is finished. Next we will prove the Proposition.

(1) In each service period $t$, the optimal actions of capacity allocation is determined by the following operators:

$$ \begin{aligned} & H_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right) = \\ & \quad \left\{ {\begin{array}{*{20}l} {\max \left[ {V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) + r_{c} ,V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right) + r_{s} } \right],{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} if{\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1,s_{t} \ge 1,} \hfill \\ {V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) + r_{c} ,{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} if{\kern 1pt} c_{{t,rt_{0} }} \ge 1,s_{t} = 0,} \hfill \\ {V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right) + r_{s} ,{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} if{\kern 1pt} c_{{t,rt_{0} }} = 0,s_{t} \ge 1,} \hfill \\ {V_{t}^{\alpha } \left( {0,0} \right),{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} if{\kern 1pt} c_{{t,rt_{0} }} = s_{t} = 0.} \hfill \\ \end{array} } \right. \\ \end{aligned} $$

When both types of patients are waiting in the queue, the optimal choice is influenced by the parameters of the two types and the current revenue of different actions. In the case of $c_{{t,rt_{0} }} \ge 1$ and $s_{t} \ge 1$, the scheduled patient will be selected for surgery if and only if $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) + r_{c} \le V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right) + r_{s}$, that is $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right) \le r_{s} - r_{c}$. From Eq. (2), it can be seen that $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right)$ is a decreasing function of $c_{{t,rt_{0} }}$, so let $\Delta V_{t}^{\alpha } = V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right)$, $c_{t}^{ * } \left( s \right)$ is introduced and defined as:

$$c_{t}^{ * } \left( s \right) = \left\{ {\begin{array}{ll} {\min \left( {c_{{t,rt_{0} }} |V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) < r_{c} - r_{s} } \right),}&{\Delta V_{t}^{\alpha } < r_{c} - r_{s} } \\ {t + 1,}&{ \Delta V_{t}^{\alpha } \ge r_{c} - r_{s} } \\ \end{array} } \right.$$

. Thus, the scheduled patient will be selected for surgery if and only if $c < c_{t}^{ * } \left( s \right)$.

(2) Then $c_{t}^{ * } \left( s \right)$ and $c_{t}^{ * } \left( {s + 1} \right)$ are compared. From Eq. (3) and the definition of $c_{t}^{ * } \left( s \right)$, we can get the following formula:

$$ V_{t}^{\alpha } \left( {c_{t}^{ * } \left( {s + 1} \right) + 1,s_{t} - 1} \right) - V_{t}^{\alpha } \left( {c_{t}^{ * } \left( {s + 1} \right),s_{t} } \right) \ge V_{t}^{\alpha } \left( {c_{t}^{ * } \left( {s + 1} \right) + 1,s_{t} } \right) - V_{t}^{\alpha } \left( {c_{t}^{ * } \left( {s + 1} \right),s_{t} + 1} \right) $$

That is $V_{t}^{\alpha } \left( {c_{t}^{ * } \left( {s + 1} \right) + 1,s_{t} - 1} \right) - V_{t}^{\alpha } \left( {c_{t}^{ * } \left( {s + 1} \right),s_{t} } \right) \ge r_{c} - r_{s}$, so $c_{t}^{ * } \left( s \right) \le c_{t}^{ * } \left( {s + 1} \right)$.

Proof of Proposition 3

In order to prove that ‘the critical index $c_{t}^{ * } \left( s \right)$ is a nonincreasing function of $k_{c}$ when the penalty function is the linear form $f\left( {c_{t} ,s_{t} } \right) = - c_{t} \cdot k_{c} - s_{t} \cdot k_{s}$’, two different values of penalty cost $k_{c}$ and $\tilde{k}_{c}$ of convalescent patients are introduced, and $k_{c} < \tilde{k}_{c}$. In this case, $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right)$ and $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right)$ are the optimal reward functions with the different penalty cost. The critical index $c_{t}^{ * } \left( s \right)$ is a nonincreasing function of $k_{c}$ if the following formula:

$$ \begin{aligned} &V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,k_{c} } \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,k_{c} } \right) \le V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,\tilde{k}_{c} } \right) \\ & \quad- V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,\tilde{k}_{c} } \right) \end{aligned}$$

is satisfied for each service period $t$ and each state $\left( {c_{{t,rt_{0} }} ,s_{t} } \right) \in S_{t}$.

When $t = T + 1$,

$$ \begin{gathered} V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,k_{c} } \right) - V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,k_{c} } \right) - V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,\tilde{k}_{c} } \right) \hfill \\\quad + V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,\tilde{k}_{c} } \right) = k_{c} - \tilde{k}_{c} < 0 \hfill \\ \end{gathered} $$

For other service period $t = 1,2, \ldots ,T$,

$$ \begin{aligned} & V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,k_{c} } \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,k_{c} } \right) \\& \quad- V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,\tilde{k}_{c} } \right) + V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,\tilde{k}_{c} } \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ \begin{gathered} H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} ,k_{c} } \right) \hfill \\ - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} ,\tilde{k}_{c} } \right) \hfill \\ \end{gathered} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ \begin{gathered} H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,k_{c} } \right) \hfill \\ - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,\tilde{k}_{c} } \right) \hfill \\ \end{gathered} \right] \\ \end{aligned} $$

different conditions are considered respectively, and $a_{t + 1}$ is assumed to be 0.

In this condition, there are two expressions in the above equation, the second one is explained detailed as the following, while the first one share the similar structure.

(1) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} \ge 1$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,k_{c} } \right) \\ & \quad - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,\tilde{k}_{c} } \right) \\ & \quad = \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,k_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,k_{c} } \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,\tilde{k}_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s} } \right] \\ \end{aligned} $$

(1) If $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,k_{c} } \right) + r_{c} \ge V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{s}$ and $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{c} \ge V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,k_{c} } \right) \\ & \quad - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,\tilde{k}_{c} } \right) \\ & \quad = V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,k_{c} } \right) + r_{c} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{c} \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,k_{c} } \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,\tilde{k}_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{s} } \right] \\ & \quad \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,k_{c} } \right) + r_{c} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{c} \\ & \quad \quad - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) - r_{c} - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,\tilde{k}_{c} } \right) - r_{c} \le 0 \\ \end{aligned} $$

(2) If $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,k_{c} } \right) + r_{c} \ge V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{s}$ and $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{c} \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,k_{c} } \right) \\ & \quad - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,\tilde{k}_{c} } \right) \\ & \quad = V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,k_{c} } \right) + r_{c} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s} \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,k_{c} } \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,\tilde{k}_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{s} } \right] \\ & \quad \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,k_{c} } \right) + r_{c} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s} \\ & \quad \quad - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,k_{c} } \right) - r_{s} - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,\tilde{k}_{c} } \right) - r_{c} \\ \end{aligned} $$

It is known that

$$\begin{aligned} &V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,k_{c} } \right) - V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) \\&\quad- V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,\tilde{k}_{c} } \right) + V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) \le 0 \end{aligned}$$

$$\begin{aligned} &V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) - V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,k_{c} } \right) \\&\quad- V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,\tilde{k}_{c} } \right) \le 0 \end{aligned}$$

By adding up the above two formulas, the following formula is got:

$$ \begin{aligned} & V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,k_{c} } \right) - V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,k_{c} } \right) \\ & \quad - V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,\tilde{k}_{c} } \right) + V_{T + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,\tilde{k}_{c} } \right) \le 0 \\ \end{aligned} $$

So,

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,k_{c} } \right) \\&\quad - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,\tilde{k}_{c} } \right) \\ & \quad \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,k_{c} } \right) + r_{c} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s} \\ & \quad \quad - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,k_{c} } \right) - r_{s} - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,\tilde{k}_{c} } \right) - r_{c} \le 0 \\ \end{aligned} $$

3) If $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,k_{c} } \right) + r_{c} \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{s}$ and $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{c} \ge V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,k_{c} } \right) \\&\quad - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,\tilde{k}_{c} } \right) \\ & \quad = V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{s} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{c} \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,k_{c} } \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,\tilde{k}_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{s} } \right] \\ & \quad \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{s} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{c} \\ & \quad \quad - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) - r_{c} - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) - r_{s} = 0 \\ \end{aligned} $$

4) If $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,k_{c} } \right) + r_{c} \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{s}$ and $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{c} \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,k_{c} } \right) \\&\quad - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,\tilde{k}_{c} } \right) \\ & \quad = V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{s} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s} \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,k_{c} } \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1,\tilde{k}_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) + r_{s} } \right] \\ & \quad \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,k_{c} } \right) + r_{s} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s} \\ & \quad \quad - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1,k_{c} } \right) - r_{s} - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} ,\tilde{k}_{c} } \right) - r_{s} \le 0 \\ \end{aligned} $$

(2) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} = 0$

$$ \begin{gathered} H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0,\tilde{k}_{c} } \right) \hfill \\ = \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1,k_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0,k_{c} } \right) + r_{s} } \right] \hfill \\ - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1,\tilde{k}_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0,\tilde{k}_{c} } \right) + r_{s} } \right] \hfill \\ - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0,k_{c} } \right) - r_{c} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0,\tilde{k}_{c} } \right) + r_{c} \hfill \\ \end{gathered} $$

1) If $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1,k_{c} } \right) + r_{c} \ge V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0,k_{c} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0,\tilde{k}_{c} } \right) \\ & \quad \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1,k_{c} } \right) + r_{c} - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1,\tilde{k}_{c} } \right) - r_{c} \\ & \quad \quad - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0,k_{c} } \right) - r_{c} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0,\tilde{k}_{c} } \right) + r_{c} \le 0 \\ \end{aligned} $$

2) If $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1,k_{c} } \right) + r_{c} \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0,k_{c} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0,\tilde{k}_{c} } \right) \\ & \quad \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0,k_{c} } \right) + r_{s} - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0,\tilde{k}_{c} } \right) - r_{s} \\ & \quad \quad - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0,k_{c} } \right) - r_{c} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0,\tilde{k}_{c} } \right) + r_{c} = 0 \\ \end{aligned} $$

(3) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} \ge 1$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {0,s_{t} + 1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} ,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} ,\tilde{k}_{c} } \right) \\ & \quad = V_{t + 1}^{\alpha } \left( {0,s_{t} ,k_{c} } \right) + r_{s} - V_{t + 1}^{\alpha } \left( {0,s_{t} ,\tilde{k}_{c} } \right) - r_{s} \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {0,s_{t} ,k_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {1,s_{t} - 1,k_{c} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {V_{t + 1}^{\alpha } \left( {0,s_{t} ,\tilde{k}_{c} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s} } \right] \\ \end{aligned} $$

1) If $V_{t + 1}^{\alpha } \left( {0,s_{t} ,\tilde{k}_{c} } \right) + r_{c} \ge V_{t + 1}^{\alpha } \left( {1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {0,s_{t} + 1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} ,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} ,\tilde{k}_{c} } \right) \\ & \quad \le V_{t + 1}^{\alpha } \left( {0,s_{t} ,k_{c} } \right) + r_{s} - V_{t + 1}^{\alpha } \left( {0,s_{t} ,\tilde{k}_{c} } \right) - r_{s} \\ & \quad \quad + V_{t + 1}^{\alpha } \left( {0,s_{t} ,\tilde{k}_{c} } \right) + r_{c} - V_{t + 1}^{\alpha } \left( {0,s_{t} ,k_{c} } \right) - r_{c} = 0 \\ \end{aligned} $$

2) If $V_{t + 1}^{\alpha } \left( {0,s_{t} ,\tilde{k}_{c} } \right) + r_{c} \le V_{t + 1}^{\alpha } \left( {1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {0,s_{t} + 1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} ,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} ,\tilde{k}_{c} } \right) \\ & \quad = V_{t + 1}^{\alpha } \left( {0,s_{t} ,k_{c} } \right) + r_{s} - V_{t + 1}^{\alpha } \left( {0,s_{t} ,\tilde{k}_{c} } \right) - r_{s} \\ & \quad \quad + V_{t + 1}^{\alpha } \left( {1,s_{t} - 1,\tilde{k}_{c} } \right) + r_{s} - V_{t + 1}^{\alpha } \left( {1,s_{t} - 1,k_{c} } \right) - r_{s} \le 0 \\ \end{aligned} $$

(4) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} = 0$

$$ \begin{aligned} & H_{t + 1}^{\alpha } \left( {0,1,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {1,0,k_{c} } \right) - H_{t + 1}^{\alpha } \left( {0,1,\tilde{k}_{c} } \right) + H_{t + 1}^{\alpha } \left( {1,0,\tilde{k}_{c} } \right) \\ & \quad = V_{t + 1}^{\alpha } \left( {0,0,k_{c} } \right) + r_{s} - V_{t + 1}^{\alpha } \left( {0,0,k_{c} } \right) - r_{c} \\ & \quad \quad - V_{t + 1}^{\alpha } \left( {0,0,\tilde{k}_{c} } \right) - r_{s} + V_{t + 1}^{\alpha } \left( {0,0,\tilde{k}_{c} } \right) + r_{c} = 0 \\ \end{aligned} $$

Thus, from the above, we can get the conclusion that:

$$\begin{aligned} V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,k_{c} } \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,k_{c} } \right) &\le V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1,\tilde{k}_{c} } \right) \\ & \quad- V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} ,\tilde{k}_{c} } \right) \end{aligned}$$

So, ‘the critical index $c_{t}^{ * } \left( s \right)$ is a nondecreasing function of $k_{s}$’ can be proved in the similar way.

Proof of Proposition 4

$G_{it} ,i = 1,2,3$ is defined as the classes of profit functions on state $S_{t}$. For each $\left( {c_{{t,rt_{0} }} ,s_{t} } \right) \in S_{t}$, and ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$, $s_{t} \ge 1$, $t = 1,2, \ldots ,T + 1$, we define the following three conditions. For each $g\left( {c_{{rt_{0} }} ,s} \right) \in G_{1t}$, $g\left( {c_{{rt_{0} }} - 1,s} \right) - g\left( {c_{{rt_{0} }} ,s - 1} \right) \le r_{s} - r_{c}$; for each $g\left( {c_{{rt_{0} }} ,s} \right) \in G_{2t}$, $g\left( {c_{{rt_{0} }} - 1,s} \right) - g\left( {c_{{rt_{0} }} ,s - 1} \right) \ge r_{s} - r_{c}$; for each $g\left( {c_{{rt_{0} }} ,s} \right) \in G_{3t}$, $g\left( {c_{{rt_{0} }} - 1,s} \right) - g\left( {c_{{rt_{0} }} ,s - 1} \right) = g\left( {c_{{rt_{0} }} - 1,s + 1} \right) - g\left( {c_{{rt_{0} }} ,s} \right)$.

It can be observed that: if the optimal reward function $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right)$ belongs to $G_{1t}$, the optimal choice is to serve scheduled patients at the $t$ th service period; if the optimal reward function $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right)$ belongs to $G_{2t}$, the optimal choice is to serve convalescent patients at the $t$ th service period; while the optimal reward function $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right)$ belongs to $G_{3t}$, the optimal choice on which type of patient to serve at the $t$ th service period is influenced by the number of convalescent patients waiting in the line.

By this equation $H_{T + 1}^{\alpha } \left( {c_{{T + 1,rt_{0} }} ,s_{T + 1} } \right) = V_{T + 1}^{\alpha } \left( {c_{{T + 1,rt_{0} }} ,s_{T + 1} } \right) = f\left( {c_{{T + 1,rt_{0} }} ,s_{T + 1} } \right)$, and $a_{t + 1}$ is assumed to be 0, then the following formula is got.

(1) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} \ge 1$

$$ \begin{aligned} & V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - V_{T - 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + V_{T - 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \\ & \quad = \left[ { - \left( {s_{t} + 1} \right) \cdot \left( {w_{s} + k_{s} } \right) - c_{{t,rt_{0} }} \cdot \left( {w_{c} + k_{c} } \right) - p_{{t,rt_{1} }} \cdot k_{c} } \right] \\ & \quad \quad - \left[ { - s_{t} \cdot \left( {w_{s} + k_{s} } \right) - \left( {c_{{t,rt_{0} }} + 1} \right) \cdot \left( {w_{c} + k_{c} } \right) - p_{{t,rt_{1} }} \cdot k_{c} } \right] \\ & \quad \quad - \left[ { - \left( {s_{t} + 1} \right) \cdot w_{s} - c_{{t,rt_{0} }} \cdot w_{c} + p_{{t,rt_{1} }} \cdot H_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) }\right.\\ & \quad \quad \left.{- \left( {1 - p_{{t,rt_{1} }} } \right) \cdot H_{T}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right)} \right] \\ & \quad \quad + \left[ { - s_{t} \cdot w_{s} - \left( {c_{{t,rt_{0} }} + 1} \right) \cdot w_{c} + p_{{t,rt_{1} }} \cdot H_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right)}\right.\\ & \quad \quad \left.{ + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot H_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right)} \right] \\ & \quad \quad = k_{c} - k_{s} - p_{{t,rt_{1} }} \cdot \left[ {H_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right)} \right] \\ & \quad \quad - \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{T}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - H_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right)} \right] \\ & \quad = k_{c} - k_{s} - p_{{t,rt_{1} }} \cdot \left[ {V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right)} \right] \\ & \quad \quad - \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1} \right) - V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right)} \right] \\ & \quad = k_{c} - k_{s} - p_{{t,rt_{1} }} \cdot \left( {w_{c} + k_{c} - w_{s} - k_{s} } \right) - \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left( {w_{c} + k_{c} - w_{s} - k_{s} } \right) \\ & \quad = w_{s} - w_{c} \ge 0 \\ \end{aligned} $$

(2) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} = 0$

$$ \begin{aligned} & V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) - V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) - V_{T - 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) + V_{T - 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) \\ & \quad = k_{c} - k_{s} - p_{{t,rt_{1} }} \cdot \left[ {H_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) - H_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right)} \right] \\ & \quad \quad - \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{T}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) - H_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right)} \right] \\ & \quad = k_{c} - k_{s} - p_{{t,rt_{1} }} \cdot \left[ {V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) - V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right)} \right] \\ & \quad \quad - \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1} \right) - V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right)} \right] \\ & \quad = w_{s} - w_{c} \ge 0 \\ \end{aligned} $$

(3) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} \ge 1$

$$ \begin{aligned} & V_{T}^{\alpha } \left( {0,s_{t} + 1} \right) - V_{T}^{\alpha } \left( {1,s_{t} } \right) - V_{T - 1}^{\alpha } \left( {0,s_{t} + 1} \right) + V_{T - 1}^{\alpha } \left( {1,s_{t} } \right) \\ & \quad = k_{c} - k_{s} - p_{{t,rt_{1} }} \cdot \left[ {H_{T}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{T}^{\alpha } \left( {2,s_{t} } \right)} \right]\\&\quad - \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{T}^{\alpha } \left( {0,s_{t} + 1} \right) - H_{T}^{\alpha } \left( {1,s_{t} } \right)} \right] \\ & \quad = p_{{t,rt_{1} }} \cdot \left( {w_{s} - w_{c} } \right) + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left( {w_{c} + r_{c} - r_{s} - k_{s} } \right) \ge 0 \\ \end{aligned} $$

(4) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} = 0$

$$ \begin{aligned} & V_{T}^{\alpha } \left( {0,1} \right) - V_{T}^{\alpha } \left( {1,0} \right) - V_{T - 1}^{\alpha } \left( {0,1} \right) + V_{T - 1}^{\alpha } \left( {1,0} \right) \\ & \quad = k_{c} - k_{s} - p_{{t,rt_{1} }} \cdot \left[ {H_{T}^{\alpha } \left( {1,1} \right) - H_{T}^{\alpha } \left( {2,0} \right)} \right] - \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{T}^{\alpha } \left( {0,1} \right) - H_{T}^{\alpha } \left( {1,0} \right)} \right] \\ & \quad = k_{c} - k_{s} - p_{{t,rt_{1} }} \cdot \left( {w_{c} + k_{c} - w_{s} - k_{s} } \right) - \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left( {r_{s} - r_{c} } \right) \\ & \quad = \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left( {r_{c} + k_{c} - r_{s} - k_{s} } \right) + p_{{t,rt_{1} }} \cdot \left( {w_{s} - w_{c} } \right) \ge 0 \\ \end{aligned} $$

So, $V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - V_{T}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - V_{T - 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + V_{T - 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \ge 0$. Then for other service period $t$ and each state $\left( {c_{{t,rt_{0} }} ,s_{t} } \right) \in S$, we want to show that $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \ge 0$. 4 different conditions are considered respectively and $a_{t + 1}$ is assumed to be 0.

(1) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} \ge 1$

$$ \begin{aligned} & V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right)}\right.\\& \quad \quad\left.{ - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) }\right.\\& \quad \quad\left.{- H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right)} \right] \\ \end{aligned} $$

In this condition, there are two expressions, the first one is explained detailed as the following, while the second one share the similar structure.

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = \max \left[ {V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ \end{aligned} $$

1) If $V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} \ge V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s}$, and $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} \ge V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = \max \left[ {V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - r_{c} - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - r_{c} \\ & \quad \ge V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \ge 0 \\ \end{aligned} $$

2) If $V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} \ge V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s}$, and $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = \max \left[ {V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - r_{c} - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - r_{s} \\ & \quad \ge V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) = 0 \\ \end{aligned} $$

3) If $V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} \le V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s}$, and $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} \ge V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = \max \left[ {V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) - r_{s} - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - r_{c} \\ & \quad \ge V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right)\\& \quad \quad - V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) \ge 0 \\ \end{aligned} $$

4) If $V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} \le V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s}$, and $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s}$

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) - H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} } \right) \\ & \quad = \max \left[ {V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad - V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) - r_{s} - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - r_{s} \\ & \quad \ge V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) \\&\quad \quad- V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,s_{t} - 1} \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \ge 0 \\ \end{aligned} $$

(2) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} = 0$

$$ \begin{aligned} & V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) + V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) - H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right) }\right.\\& \quad \quad \left.{- H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 2,0} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) - H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) }\right.\\& \quad \quad \left.{- H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right)} \right] \\ \end{aligned} $$

In this condition, there are two expressions, the second one is explained detailed as the following, while the first one share the similar structure.

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) - H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) \\ & \quad = \max \left[ {V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1} \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1} \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) + r_{s} } \right] \\ & \quad \quad - V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) - r_{c} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) + r_{c} \\ \end{aligned} $$

1) If $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1} \right) + r_{c} \ge V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) - H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) \\ & \quad = \max \left[ {V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1} \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) + r_{s} } \right] \\ & \quad \quad - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1} \right) - r_{c} - V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) - r_{c} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) + r_{c} \\ & \quad \ge V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1} \right) + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1} \right) - V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) \ge 0 \\ \end{aligned} $$

2) If $V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1} \right) + r_{c} \le V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) - H_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,1} \right) + H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,0} \right) \\ & \quad = \max \left[ {V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,1} \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) + r_{s} } \right] \\ & \quad \quad - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) - r_{s} - V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) - r_{c} + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) + r_{c} \\ & \quad \ge V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) + V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) - V_{t + 2}^{\alpha } \left( {c_{{t,rt_{0} }} ,0} \right) = 0 \\ \end{aligned} $$

(3) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} \ge 1$

$$ \begin{aligned} & V_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) - V_{t + 1}^{\alpha } \left( {1,s_{t} } \right) - V_{t}^{\alpha } \left( {0,s_{t} + 1} \right) + V_{t}^{\alpha } \left( {1,s_{t} } \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 2}^{\alpha } \left( {1,s_{t} + 1} \right) - H_{t + 2}^{\alpha } \left( {2,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {1,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {2,s_{t} } \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 2}^{\alpha } \left( {0,s_{t} + 1} \right) - H_{t + 2}^{\alpha } \left( {1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} } \right)} \right] \\ \end{aligned} $$

In this condition, there are two expressions, the second one is explained detailed as the following, while the first expression’s prove is same as the case of ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 1$ and $s_{t} \ge 1$.

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {0,s_{t} + 1} \right) - H_{t + 2}^{\alpha } \left( {1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) \\ & \quad = V_{t + 2}^{\alpha } \left( {0,s_{t} } \right) + r_{s} - V_{t + 1}^{\alpha } \left( {0,s_{t} } \right) - r_{s} \\ & \quad \quad - \max \left[ {V_{t + 2}^{\alpha } \left( {0,s_{t} } \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \quad + \max \left[ {V_{t + 1}^{\alpha } \left( {0,s_{t} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {1,s_{t} - 1} \right) + r_{s} } \right] \\ \end{aligned} $$

1) If $V_{t + 2}^{\alpha } \left( {0,s_{t} } \right) + r_{c} \ge V_{t + 2}^{\alpha } \left( {1,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {0,s_{t} + 1} \right) - H_{t + 2}^{\alpha } \left( {1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) \\ & \quad = V_{t + 2}^{\alpha } \left( {0,s_{t} } \right) + r_{s} - V_{t + 1}^{\alpha } \left( {0,s_{t} } \right) - r_{s} - V_{t + 2}^{\alpha } \left( {0,s_{t} } \right) - r_{c} \\ & \quad \quad + \max \left[ {V_{t + 1}^{\alpha } \left( {0,s_{t} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \ge V_{t + 2}^{\alpha } \left( {0,s_{t} } \right) + V_{t + 1}^{\alpha } \left( {0,s_{t} } \right) - V_{t + 1}^{\alpha } \left( {0,s_{t} } \right) - V_{t + 2}^{\alpha } \left( {0,s_{t} } \right) = 0 \\ \end{aligned} $$

2) If $V_{t + 2}^{\alpha } \left( {0,s_{t} } \right) + r_{c} \le V_{t + 2}^{\alpha } \left( {1,s_{t} - 1} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {0,s_{t} + 1} \right) - H_{t + 2}^{\alpha } \left( {1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) + H_{t + 1}^{\alpha } \left( {1,s_{t} } \right) \\ & \quad = V_{t + 2}^{\alpha } \left( {0,s_{t} } \right) + r_{s} - V_{t + 1}^{\alpha } \left( {0,s_{t} } \right) - r_{s} - V_{t + 2}^{\alpha } \left( {1,s_{t} - 1} \right) - r_{s} \\ & \quad \quad + \max \left[ {V_{t + 1}^{\alpha } \left( {0,s_{t} } \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {1,s_{t} - 1} \right) + r_{s} } \right] \\ & \quad \ge V_{t + 1}^{\alpha } \left( {1,s_{t} - 1} \right) + V_{t + 2}^{\alpha } \left( {0,s_{t} } \right) - V_{t + 1}^{\alpha } \left( {0,s_{t} } \right) - V_{t + 2}^{\alpha } \left( {1,s_{t} - 1} \right) \ge 0 \\ \end{aligned} $$

(4) ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 0$ and $s_{t} = 0$

$$ \begin{aligned} & V_{t + 1}^{\alpha } \left( {0,1} \right) - V_{t + 1}^{\alpha } \left( {1,0} \right) - V_{t}^{\alpha } \left( {0,1} \right) + V_{t}^{\alpha } \left( {1,0} \right) \\ & \quad = p_{{t,rt_{1} }} \cdot \left[ {H_{t + 2}^{\alpha } \left( {1,1} \right) - H_{t + 2}^{\alpha } \left( {2,0} \right) - H_{t + 1}^{\alpha } \left( {1,1} \right) + H_{t + 1}^{\alpha } \left( {2,0} \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {H_{t + 2}^{\alpha } \left( {0,1} \right) - H_{t + 2}^{\alpha } \left( {1,0} \right) - H_{t + 1}^{\alpha } \left( {0,1} \right) + H_{t + 1}^{\alpha } \left( {1,0} \right)} \right] \\ \end{aligned} $$

In this condition, there are two expressions, the first one is explained detailed as the following, while the second one share the similar structure.

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {1,1} \right) - H_{t + 2}^{\alpha } \left( {2,0} \right) - H_{t + 1}^{\alpha } \left( {1,1} \right) + H_{t + 1}^{\alpha } \left( {2,0} \right) \\ & \quad = \max \left[ {V_{t + 2}^{\alpha } \left( {0,1} \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - \max \left[ {V_{t + 1}^{\alpha } \left( {0,1} \right) + r_{c} ,V_{t + 1}^{\alpha } \left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - V_{t + 2}^{\alpha } \left( {1,0} \right) - r_{c} + V_{t + 1}^{\alpha } \left( {1,0} \right) + r_{c} \\ \end{aligned} $$

1) If $V_{t + 1}^{\alpha } \left( {0,1} \right) + r_{c} \ge V_{t + 1}^{\alpha } \left( {1,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {1,1} \right) - H_{t + 2}^{\alpha } \left( {2,0} \right) - H_{t + 1}^{\alpha } \left( {1,1} \right) + H_{t + 1}^{\alpha } \left( {2,0} \right) \\ & \quad = \max \left[ {V_{t + 2}^{\alpha } \left( {0,1} \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - V_{t + 1}^{\alpha } \left( {0,1} \right) - r_{c} - V_{t + 2}^{\alpha } \left( {1,0} \right) - r_{c} + V_{t + 1}^{\alpha } \left( {1,0} \right) + r_{c} \\ & \quad \ge V_{t + 2}^{\alpha } \left( {0,1} \right) + V_{t + 1}^{\alpha } \left( {1,0} \right) - V_{t + 1}^{\alpha } \left( {0,1} \right) - V_{t + 2}^{\alpha } \left( {1,0} \right) \ge 0 \\ \end{aligned} $$

2) If $V_{t + 1}^{\alpha } \left( {0,1} \right) + r_{c} \le V_{t + 1}^{\alpha } \left( {1,0} \right) + r_{s}$

$$ \begin{aligned} & H_{t + 2}^{\alpha } \left( {1,1} \right) - H_{t + 2}^{\alpha } \left( {2,0} \right) - H_{t + 1}^{\alpha } \left( {1,1} \right) + H_{t + 1}^{\alpha } \left( {2,0} \right) \\ & \quad = \max \left[ {V_{t + 2}^{\alpha } \left( {0,1} \right) + r_{c} ,V_{t + 2}^{\alpha } \left( {1,0} \right) + r_{s} } \right] \\ & \quad \quad - V_{t + 1}^{\alpha } \left( {1,0} \right) - r_{s} - V_{t + 2}^{\alpha } \left( {1,0} \right) - r_{c} + V_{t + 1}^{\alpha } \left( {1,0} \right) + r_{c} \\ & \quad \ge V_{t + 2}^{\alpha } \left( {1,0} \right) + V_{t + 1}^{\alpha } \left( {1,0} \right) - V_{t + 1}^{\alpha } \left( {1,0} \right) - V_{t + 2}^{\alpha } \left( {1,0} \right) = 0 \\ \end{aligned} $$

From the above, we can get the conclusion that:$V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) + V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \ge 0$.

In the second part, we will show the proof of ‘There exists a critical index $c_{t}^{ * } \left( s \right)$, and $c_{t + 1}^{ * } \left( s \right) \le c_{t}^{ * } \left( s \right)$’.

At the $t$ th service period, when $t = T$, the following equation is got: $V_{T}^{\alpha } \left( {c_{{T,rt_{0} }} - 1,s_{T} } \right) - V_{T}^{\alpha } \left( {c_{{T,rt_{0} }} ,s_{T} - 1} \right) = w_{c} + k_{c} - w_{s} - k_{s} \ge r_{s} - r_{c} ,\forall \left( {c_{{t,rt_{0} }} ,s_{t} } \right) \in S_{T}$. So, for ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 0$, $V_{T}^{\alpha } \left( {c_{{T,rt_{0} }} ,s_{T} } \right) \in G_{2t}$, $V_{T}^{\alpha } \left( {c_{{T,rt_{0} }} ,s_{T} } \right) \in G_{3t}$, and $c_{t}^{ * } \left( s \right)$ can be set to 0.

At the $t + 1$ th service period, if ${\kern 1pt} {\kern 1pt} c_{{t + 1,rt_{0} }} \ge c_{t + 1}^{ * } \left( s \right)$, $V_{t + 1}^{\alpha } \left( {c_{{t + 1,rt_{0} }} ,s_{t + 1} } \right) \in G_{2,t + 1} ,G_{3,t + 1}$; if ${\kern 1pt} {\kern 1pt} c_{{t + 1,rt_{0} }} < c_{t + 1}^{ * } \left( s \right)$, $V_{t + 1}^{\alpha } \left( {c_{{t + 1,rt_{0} }} ,s_{t + 1} } \right) \in G_{1,t + 1}$. In order to prove that $c_{t + 1}^{ * } \left( s \right) \le c_{t}^{ * } \left( s \right)$ is satisfied in each $t$ th service period, we define $U_{t + 1} \left( {c_{{t + 1,rt_{0} }} ,s_{t + 1} } \right) = V_{t + 1}^{\alpha } \left( {c_{{t + 1,rt_{0} }} - 1,s_{t + 1} } \right) - V_{t + 1}^{\alpha } \left( {c_{{t + 1,rt_{0} }} ,s_{t + 1} - 1} \right)$, for ${\kern 1pt} {\kern 1pt} c_{{t + 1,rt_{0} }} > c_{t + 1}^{ * } \left( s \right)$,

$$ \begin{aligned} & V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right) \\ & \quad = - w_{s} + w_{c} + p_{{t,rt_{1} }} \cdot \left[ {\left( {1 - p_{s} \cdot a_{t + 1} } \right) \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right) + p_{s} \cdot a_{t + 1} \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right)} \right] \\ & \quad \quad - p_{{t,rt_{1} }} \cdot \left[ {\left( {1 - p_{s} \cdot a_{t + 1} } \right) \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) + p_{s} \cdot a_{t + 1} \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right)} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {\left( {1 - p_{s} \cdot a_{t + 1} } \right) \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) + p_{s} \cdot a_{t + 1} \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1} \right)} \right] \\ & \quad - \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ {\left( {1 - p_{s} \cdot a_{t + 1} } \right) \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right) + p_{s} \cdot a_{t + 1} \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right)} \right] \\ \\ \end{aligned} $$

$a_{t + 1}$ is assumed to be 1, so there is:

$$ \begin{aligned} & V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right) \\ & \quad = - w_{s} + w_{c} + p_{{t,rt_{1} }} \cdot \left[ \begin{gathered} \left( {1 - p_{s} } \right) \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} - 1} \right) \hfill \\ + p_{s} \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} + 1,s_{t} } \right) \hfill \\ \end{gathered} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ \begin{gathered} \left( {1 - p_{s} } \right) \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right) \hfill \\ + p_{s} \cdot H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1} \right) - H_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right) \hfill \\ \end{gathered} \right] \\ \end{aligned} $$

If ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 2$, then

$$ \begin{aligned} & V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right) - r_{s} + r_{c} \\ & \quad = - w_{s} + w_{c} + p_{{t,rt_{1} }} \cdot \left[ \begin{gathered} \left( {1 - p_{s} } \right) \cdot \left( {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right)} \right) \hfill \\ + p_{s} \cdot \left( {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} + 1} \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right)} \right) \hfill \\ \end{gathered} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ \begin{gathered} \left( {1 - p_{s} } \right) \cdot \left( {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 2,s_{t} } \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} - 1} \right)} \right) \hfill \\ + p_{s} \cdot \left( {V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 2,s_{t} + 1} \right) - V_{t + 1}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right)} \right) \hfill \\ \end{gathered} \right] - r_{s} + r_{c} \\ & \quad = - w_{s} + w_{c} + p_{{t,rt_{1} }} \cdot U_{t + 1} \left( {c_{{t + 1,rt_{0} }} ,s_{t + 1} } \right) \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot U_{t + 1} \left( {c_{{t + 1,rt_{0} }} - 1,s_{t + 1} } \right) - r_{s} + r_{c} \\ \end{aligned} $$

If ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 1$, then

$$ \begin{aligned} & V_{t}^{\alpha } \left( {0,s_{t} } \right) - V_{t}^{\alpha } \left( {1,s_{t} - 1} \right) - r_{s} + r_{c} \\ & \quad = - w_{s} + w_{c} + p_{{t,rt_{1} }} \cdot \left[ \begin{gathered} \left( {1 - p_{s} } \right) \cdot \left( {V_{t + 1}^{\alpha } \left( {0,s_{t} } \right) - V_{t + 1}^{\alpha } \left( {1,s_{t} - 1} \right)} \right) \hfill \\ + p_{s} \cdot \left( {V_{t + 1}^{\alpha } \left( {0,s_{t} + 1} \right) - V_{t + 1}^{\alpha } \left( {1,s_{t} } \right)} \right) \hfill \\ \end{gathered} \right] \\ & \quad \quad + \left( {1 - p_{{t,rt_{1} }} } \right) \cdot \left[ \begin{gathered} \left( {1 - p_{s} } \right) \cdot \left( {V_{t + 1}^{\alpha } \left( {0,s_{t} - 1} \right) + r_{s} - V_{t + 1}^{\alpha } \left( {0,s_{t} - 1} \right) - r_{c} } \right) \hfill \\ + p_{s} \cdot \left( {V_{t + 1}^{\alpha } \left( {0,s_{t} } \right) + r_{s} - V_{t + 1}^{\alpha } \left( {0,s_{t} } \right) - r_{c} } \right) \hfill \\ \end{gathered} \right] - r_{s} + r_{c} \\ & \quad = - w_{s} + w_{c} + p_{{t,rt_{1} }} \cdot U_{t + 1} \left( {1,s_{t + 1} } \right) + \left( {1 - p_{{t,rt_{1} }} } \right)\left( {r_{s} - r_{c} } \right) \\ \end{aligned} $$

When ${\kern 1pt} {\kern 1pt} c_{{t + 1,rt_{0} }} > c_{t + 1}^{ * } \left( s \right)$, $U_{t + 1} \left( {c_{{t + 1,rt_{0} }} - 1,s_{t + 1} } \right)$, $U_{t + 1} \left( {c_{{t + 1,rt_{0} }} ,s_{t + 1} } \right)$, and $U_{t + 1} \left( {c_{{t + 1,rt_{0} }} + 1,s_{t + 1} } \right)$ are all not depend on the number of $s_{t}$, so in the case ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} = 1$ of and ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge 2$, there is $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right) \in G_{3t}$.

From inequation (2): $g\left( {c_{{rt_{0} }} ,s + 1} \right) - g\left( {c_{{rt_{0} }} + 1,s} \right) \le g\left( {c_{{rt_{0} }} + 1,s + 1} \right) - g\left( {c_{{rt_{0} }} + 2,s} \right)$, it can be seen that $U_{t + 1} \left( {c_{{t + 1,rt_{0} }} - 1,s_{t + 1} } \right)$, $U_{t + 1} \left( {c_{{t + 1,rt_{0} }} ,s_{t + 1} } \right)$, and $U_{t + 1} \left( {c_{{t + 1,rt_{0} }} + 1,s_{t + 1} } \right)$ increase with the number of ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }}$, so for ${\kern 1pt} {\kern 1pt} c_{{t + 1,rt_{0} }} > c_{t + 1}^{ * } \left( s \right)$, there exists $c_{t}^{ * } \left( s \right)$ so that for ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} \ge c_{t}^{ * } \left( s \right)$, $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right) \in G_{2t}$, and for ${\kern 1pt} {\kern 1pt} c_{{t,rt_{0} }} < c_{t}^{ * } \left( s \right)$, $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right) \in G_{1t}$. Besides, if $w_{c} \ge w_{s}$, $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} - 1,s_{t} } \right) - V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} - 1} \right) - r_{s} + r_{c} \ge 0$ is always satisfied when $V_{t}^{\alpha } \left( {c_{{t,rt_{0} }} ,s_{t} } \right) \in G_{2t}$. Thus, in this case, convalescent patients will be served whenever there is scheduled patients waiting.