Trading Off Worst and Expected Cost in Decision Tree Problems

Abstract

We characterize the best possible trade-off achievable when optimizing the construction of a decision tree with respect to both the worst and the expected cost. It is known that a decision tree achieving the minimum possible worst case cost can behave very poorly in expectation (even exponentially worse than the optimal), and the vice versa is also true. Led by applications where deciding which optimization criterion might not be easy, several authors recently have focussed on the bicriteria optimization of decision trees. Here we sharply define the limits of the best possible trade-offs between expected and worst case cost. More precisely, we show that for every $\rho >0$ there is a decision tree D with worst testing cost at most $(1 + \rho )\textit{OPT}_W$ and expected testing cost at most $\frac{1}{1 - e^{-\rho }} \textit{OPT}_E,$ where $\textit{OPT}_W$ and $\textit{OPT}_E$ denote the minimum worst testing cost and the minimum expected testing cost of a decision tree for the given instance. We also show that this is the best possible trade-off in the sense that there are infinitely many instances for which we cannot obtain a decision tree with both worst testing cost smaller than $(1 + \rho )\textit{OPT}_W$ and expected testing cost smaller than $\frac{1}{1 - e^{-\rho }} \textit{OPT}_E$.

Appendix: Optimality of \(\mathbf {p}^*\)

Here, we show that the point $\mathbf {p}^*$ defined in (9)–(10) is an optimal solution of the NLP (5)–(8) defined in Sect. 2. For that, we use Theorem 4.3.8 of [4] that we restate here in a simplified form.

Theorem 4

Let X be an open subset of $R^n$. Consider the optimization problem P.

$$\begin{aligned} \text{ Minimize }&f(\mathbf {x}) \end{aligned}$$

(26)

$$\begin{aligned} \text{ subject } \text{ to }&g_i(\mathbf {x}) \le 0&\quad \,\,\, \text{ for } i=1,\ldots ,m \end{aligned}$$

(27)

$$\begin{aligned}&h_i(\mathbf {x}) = 0&\quad \,\,\, \text{ for } i=1,\ldots ,l \end{aligned}$$

(28)

$$\begin{aligned}&\mathbf {x} \in X \end{aligned}$$

(29)

Let $\mathbf {x}$ be a feasible solution and let $I=\{i|g_i(\mathbf {x})=0\}$. Suppose there exists scalars $u_i \ge 0$ for $i=1,\ldots ,m$ and $v_i$ for $i=1,\ldots ,l$ such that

$$\begin{aligned} - \nabla f(\mathbf {x}) = \sum _{i \in I } u_i \nabla g_i(\mathbf {x}) + \sum _{i=1}^{l} v_i \nabla h_i(\mathbf {x}) \end{aligned}$$

(30)

If f is linear, $g_i$ is convex on X for $i \in I$ and $h_i$ is linear, then $\mathbf {x}$ is an optimal solution of P.²

The non linear problem defined in (5)–(8) can be rewritten in terms of problem P. In fact, for a point $\mathbf {p}=(p_1,\ldots ,p_{C+1},z)$ we have that $f(\mathbf {p}) = - z$,

$$\begin{aligned} h_1(\mathbf {p})= & {} 1 - \sum _{i=1}^{C+1} p_i,\\ g_j(\mathbf {p})= & {} z \left( \sum _{i=1}^{C+1} i \cdot p_i \right) - \sum _{i=1}^j i \cdot p_i - (j+W) \left( \sum _{i=j+1}^{C+1} p_i \right) , \end{aligned}$$

for $j=0,\ldots ,C$ and

$$\begin{aligned} g_j(\mathbf {p}) = -p_{j-C} \end{aligned}$$

for $j=C+1,\ldots ,2C+1$.

We define an open set X that contains all feasible solutions of the problem defined in (5)–(8). The motivation is to meet the conditions of Theorem 3.3.7 of [4] that will be used to establish the convexity of $g_i$

Let

$$\begin{aligned} X= & {} \left\{ (p_1,p_2,\ldots ,p_{C+1}, z) \mid \sum _{i=1}^{C+1} p_i> 1-\frac{1}{2(C+1)} \text{ and } \right. \\&\left. p_i> -\frac{1}{2(C+1)^2} \text{ and } z > 0 \right\} . \end{aligned}$$

We have to prove that: (a) $\mathbf {p}^*$ is feasible and $I=\{0,\ldots ,C\}$; (b) f, $h_1$ and $g_i$ satisfy the conditions of Theorem 4 and (c) there are multipliers satisfying condition (30).

1.1 Feasibility of $\mathbf {p}^*$

We have that $\mathbf {p}^*$ is feasible because

$$\begin{aligned} g_j(\mathbf {p}^*)= & {} z^* \left( \sum _{i=1}^{C+1} i \cdot p^*_i \right) - \sum _{i=1}^j i \cdot p^*_i - (j+W) \left( \sum _{i=j+1}^{C+1} p^*_i \right) \\= & {} z^* \left( W-(C+W)\frac{(W-1)^C}{W^C} + (C+1)\frac{(W-1)^C}{W^C} \right) \\&~~~~~~~~~~ - \left( W - (j+W)\frac{(W-1)^j}{W^j} \right) -(j+W)\frac{(W-1)^j}{W^j} \\= & {} z^* \left( W - \frac{(W-1)^{C+1}}{W^{C+1}} \right) -W = 0 \end{aligned}$$

where the last expression holds because

$$\begin{aligned} z^* =\frac{W}{W-\frac{(W-1)^{C+1}}{W^{C+1}} } \end{aligned}$$

We have that $I=\{0,\ldots ,C\}$ because

$$\begin{aligned} g_j= -p_{j-C}= -\frac{(W-1)^{j-C-1}}{(W)^{j-C}} <0, \end{aligned}$$

for $j>C$.

1.2 Convexity of $g_j$

Because f and $h_1$ are linear and $I=\{0,\ldots ,C\}$, we only need to prove that $g_i$, for $i=0,\ldots ,C$, is convex in the open set X.

Since $g_j$, for $j=0,\ldots ,C$, is twice differentiable it follows from Theorem 3.3.7 of [4] that it is enough to show that the Hessian of $g_j$, for $j=0,\ldots ,C$, is semi-positive defined in X. In fact, the Hessian of $g_j$ is a matrix where all elements, except the first $C+1$ in the last line and the first $C+1$ in the last column, are zeros. The matrix has the structure presented below.

$$\begin{aligned} H = \left| \begin{array}{cccccccc} 0 &{} 0 &{} \cdots &{} \cdot &{} \cdot &{} 0 &{} 1 \\ 0 &{} 0 &{} \cdot &{} \cdot &{} \cdot &{} 0 &{} 2 \\ \cdot &{} \cdot &{} \cdot &{} \cdot &{} \cdot &{} \cdot &{} \cdot \\ \cdot &{} \cdot &{} \cdot &{} \cdot &{} \cdot &{} \cdot &{} \cdot \\ \cdot &{} \cdot &{} \cdot &{} \cdot &{} \cdot &{} \cdot &{} \cdot \\ 0 &{} 0 &{} \cdot &{} \cdot &{} \cdot &{} 0 &{} C+1 \\ 1 &{} 2 &{} \cdot &{} \cdot &{} \cdot &{} C+1 &{} 0 \\ \end{array} \right| \cdot \end{aligned}$$

We have that

$$\begin{aligned} \mathbf {pHp} = 2 z ( p_1 + 2p_2 + \cdots + (C+1)p_{C+1} ) \ge 0 \end{aligned}$$

for all $\mathbf {p}=(p_1,\ldots ,p_{C+1},z) \in X$ because $z>0$ and

$$\begin{aligned} \sum _{ i} i \cdot p_i \ge \sum _{ i } p_i - (C+1) \frac{1}{2(C+1)^2}> 1- \frac{1}{2(C+1)}- \frac{1}{2(C+1)}>0 \end{aligned}$$

1.3 KKT Conditions

Let $\lambda _0,\ldots ,\lambda _C$ be the dual variables associated with the constraints $g_j$, $j=0,\ldots ,C$. Let $\lambda _E$ be the dual variable associated with the constraint $\sum _{ i=1}^{C+1} p_i = 1$.

The multipliers $(\lambda _0,\ldots ,\lambda _C,\lambda _E)$ must satisfy

$$\begin{aligned} - \nabla f(\mathbf {p}^*)= & {} \sum _{i=0}^{C} \lambda _i \nabla g_i(\mathbf {p}^*) + \lambda _E \nabla h_1(\mathbf {p}^*),\\&\lambda _i \ge 0 \,\,\, \text{ for } i=0,\ldots ,C \end{aligned}$$

.

Thus, we must have

$$\begin{aligned} \sum _{j=0}^{i-1} \lambda _j (j+W) + \sum _{j=i}^{C} \lambda _j i + \lambda _{E} = z \sum _{j=0}^{C} \lambda _j i \end{aligned}$$

(31)

for $i=1,\ldots ,C+1$, and also

$$\begin{aligned} \sum _{j=0}^{C} \lambda _j \left( \sum _{i=1}^{C+1} i \cdot p^*_{i} \right) =1 \end{aligned}$$

(32)

Let

$$\begin{aligned} E= \sum _{i=1}^{C+1} i \cdot p^*_{i} \end{aligned}$$

By subtracting the Eq. (31) when $i=C$ from the same equation, when $i=C+1$, using the Eq. (32) and using the fact that $E \cdot z^* = W$, we get that $\lambda _C=\frac{1}{E^2}$. In addition, we can prove by induction that $\lambda _{j-1}=\lambda _j \frac{W-1}{W}$. For that we subtract the Eq. (31) when $i=j-1$ from the same equation, when $i=j$, and use the induction hypothesis. Thus, we get that

$$\begin{aligned} \lambda _i =\frac{(W-1)^{C-i}}{W^{C-i} E^2}, \end{aligned}$$

for every i so that all multipliers are positives.