Minimizing Energies with Hierarchical Costs

Abstract

Computer vision is full of problems elegantly expressed in terms of energy minimization. We characterize a class of energies with hierarchical costs and propose a novel hierarchical fusion algorithm. Hierarchical costs are natural for modeling an array of difficult problems. For example, in semantic segmentation one could rule out unlikely object combinations via hierarchical context. In geometric model estimation, one could penalize the number of unique model families in a solution, not just the number of models—a kind of hierarchical MDL criterion. Hierarchical fusion uses the well-known α-expansion algorithm as a subroutine, and offers a much better approximation bound in important cases.

Appendices

Appendix A: Proof of Metric Relationships

Pair (V,π) forms a tree metric if V represents an edge-weighted distance in tree π. This means that V(α,β)=d(α,β) where d(α,β) is the sum of edge weights d _ij ≥0 along a path from leaf α to leaf β. A tree metric (V,π) is therefore entirely parameterized by its edge weights d _ij where j=π(i). An r-hst metric is just a tree metric where edge costs get cheaper by a factor of $\frac{1}{r}<1$ as we descend the tree, i.e. $d_{ij} \leq\frac{1}{r} d_{jk}$ for j=π(i),k=π(j). So, r-hst metrics are a subclass of tree metrics by definition.

[Tree metrics ⊂ h-metrics]: For a tree metric to be an h-metric, d must satisfy (according to Definition 3, p. 7)

(19)

For each $i \in\mathcal{L}\cup\mathcal{S}$ use shorthand j=π(i) and consider that

(20)

(21)

(22)

(23)

Use inequalities (20) and (21) to replace the left-hand side of (19) and cancel terms with (22) and (23) to get d(α ₁,i)+d(i,α ₂)≤d(α ₁,j)+d(j,α ₂), which is clearly satisfied since d _ij ≥0. To see that some (non-h-Potts) h-metrics are not tree metrics, consider the tree and symmetric smoothness cost V(⋅,⋅) below.

[(h-Potts ∩ h-metrics) ⊈ tree metrics]: The example below is a simple h-Potts potential which is also an h-metric but is not a tree metric.

The fact that it is not a tree metric can be verified by setting up a linear program relating edge costs d _ij to node costs w _i , and noting that the system is infeasible if d _ij ≥0.

[(h-Potts with w _i ≤w _π(i)) ⊂ (h-Potts ∩ tree metrics)]: If node costs $\{w_{i}\}_{i \in\mathcal{S}\cup\{r\}}$ are non-negative and do not increase as we descend the tree (i.e. w _i ≤w _π(i)) then we can construct a tree metric by induction. Given some node $j \in\mathcal{S}\cup\{r\}$, assume we have non-negative edge costs so that, for each child $i \in\mathcal{I}(j)$, $d(\alpha,i) = \frac {1}{2}w_{i}$ for all $\alpha\in\mathcal{L}_{i}$. Then we can assign cost $d_{ij} = \frac{1}{2}(w_{j} - w_{i})$ to each child edge of j to get $d(\alpha,j) = \frac{1}{2}w_{j}$ for all $\alpha\in\mathcal{L}_{j}$. Since w _i ≤w _j we also have a tree metric for subtree j. It is not necessary to assume w _i ≤w _π(i) for an h-Potts potential to be a tree metric, as the example below demonstrates (edge costs are shown on the tree).

[(h-Potts ∩ r-hst metrics) ⊂ (h-Potts with w _i ≤w _π(i))]: As described by Kumar and Koller (2009), an r-hst metric has a constant edge cost d _ij between node j and all of its children $i \in\mathcal{I}(j)$. In other words, an r-hst metric is actually parameterized by one common ‘edge’ cost per parent node $\{d_{j}\}_{j \in\mathcal{S}\cup\{r\} }$, where $0 \leq d_{i} \leq\frac{1}{r} d_{\pi(i)}$ for all $i \in \mathcal{S}$. It is easy to see that, for an h-Potts potential to be an r-hst metric, it must have w _i =w _j −2d _j where j=π(i). Thus d _j ≥0 implies w _i ≤w _j . Also note that r>1 means quantity w _j −w _i must decrease at a rate of $\frac {1}{r}$ as we descend the tree.  □

Appendix B: Proof of Theorem 5

Proof

Without loss of generality we assume that all weights w _pq =1. Consider any local minimum $\hat{f}^{j}$ computed by h-fusion at internal node j, and let us choose some child node $i \in\mathcal{I}(j)$. We first define a useful set of pixels for i with respect to a global optimum f ^∗

$$ \mathcal{P}_i = \bigl\{ p : f_p^{*} \in\mathcal{L}_i \bigr\}. $$

This set contains all pixels assigned a label within subtree i, and so for any other child i′≠i we know that $\mathcal{P}_{i} \cap \mathcal{P}_{i'} = \emptyset$.

We can produce a labeling $\hat{f}^{j \otimes i}$ within one h-fusion move from local minimum $\hat{f}^{j}$ as follows:

$$ \hat{f}^{j \otimes i}_p = \left\{ \begin{array}{l@{\quad}l} \hat{f}^i_p & \text{if $p \in\mathcal{P}_{i}$}\\ [3pt] \hat{f}^j_p & \text{otherwise}. \end{array} \right. $$

Since each $\hat{f}^{j}$ is known to be a local optimum w.r.t. expansion moves for each $i \in\mathcal{I}(j)$ we know that

$$ E\bigl(\hat{f}^j\bigr) \leq E\bigl( \hat{f}^{j \otimes i}\bigr). $$

(24)

The general strategy to use (24) for different i to build an inequality that is ultimately of the form $E(\hat{f}^{j}) \leq E(f^{*}) + \mathrm{error}$. This will be achieved by breaking the energy terms in E into parts in such a way that a recursive inequality can be established. The recursive inequality will then be expanded until all terms can be bounded relative to E(f ^∗).

Let $E(\cdot)|_{\mathcal{A}}$ denote a restriction of the summands of energy (1) to only the following terms:

$$ E(f)|_\mathcal{A} = \sum_{p \in\mathcal{A}} D_p( f_p ) + \sum _{pq \in\mathcal{A}} V( f_p, f_q). $$

We separate the unary and pairwise terms of E(f) via interior, exterior, and boundary sets with respect to pixels $\mathcal{P}_{i}$:

Let E _H (f) denote the total label cost incurred by a labeling f, i.e. the sum of label cost terms. The following facts now hold:

(25)

(26)

We have not accounted for the label costs yet, but for simplicity we break this proof into two parts: part 1 derives the coefficient c related to smoothness costs V, and part 2 derives the coefficient c ₂ related to label costs H. For part 1 we can assume there are no label costs at all.

Part 1. Derive Coefficient c for Smoothness Cost Bound

Using (25) and (26) we can cancel out all the $\overline{\mathcal{A}}_{i}$ terms and rewrite (24) as

$$ E\bigl(\hat{f}^j\bigr)|_{\mathcal{A}_i \cup \partial\mathcal{A}_i} \leq E \bigl(\hat{f}^i\bigr)|_{\mathcal{A}_i} + E\bigl(\hat{f}^{j \otimes i}\bigr)|_{\partial\mathcal{A}_i}. $$

(27)

For each $i \in\mathcal{I}(j)$ inequality (27) contains a subset of all the energy terms in $E(\hat{f}^{j})|_{\mathcal{A}_{j}}$ pertaining to pixels $\mathcal {P}_{i}$. Let $\mathcal{I}^{*} = \{ i \in\mathcal{I}(j) : \mathcal{P}_{i} \neq \emptyset\}$ be the set of children whose sub-trees contain a label used by f ^∗. If we sum inequality (27) over all $i \in\mathcal{I}^{*}$, the left-hand side will contain all the terms in $E(\hat{f}^{j})|_{\mathcal{A}_{j}}$ (and more). Adding up all the left-hand sides we have

(28)

Using (28) and likewise adding up the right-hand sides of (27) we have

(29)

The first important observation about (29) is that each $E(\hat{f}^{i})_{\mathcal{A}_{j}}$ term on the right-hand side can be substituted by recursively applying the inequality itself. We can recursively substitute, branching further and further down the tree, until the path finally stops at a leaf $\ell\in\mathcal{L}$ giving us base case $E(\hat{f}^{\ell})|_{\mathcal{A}_{\ell}} = \sum_{p \in \mathcal{P}_{\ell}} D_{p}(f_{p}^{*})$. The sets $\{\mathcal{P}_{\ell}\}_{\ell\in\mathcal{L}}$ must be disjoint and their union is $\mathcal{P}_{j}$ so expression (29), when fully expanded, becomes roughly

(30)

The second observation about (29) is that each edge pq on an outer boundary $\partial\mathcal{A}_{i} \cap\partial \mathcal{A}_{j}$ appears once in the sum over $\mathcal{I}^{*}$ whereas each edge on an interior boundary $\partial\mathcal{A}_{i} \setminus\partial\mathcal{A}_{j}$ appears twice: once for $p \in\mathcal{A}_{i}$ and once for some $q \in\mathcal{A}_{i'}$. By careful accounting we collect all the $V(\hat{f}^{i}_{p},\hat{f}^{\pi(i)}_{q})$ terms generated by the recursive substitution and express (29) as³

(31)

where we define $\mathcal{J}(\ell;\ell')$ to be the set of nodes along the path from a label $\ell\in\mathcal{L}$ up to, but not including, the lowest common ancestor of ℓ and ℓ′, namely

All that remains is to bound each $V(\hat{f}^{i}_{p},\hat{f}^{\pi(i)}_{q})$ in terms of $V(f^{*}_{p},f^{*}_{q})$ using b _i described in Definition 7. From now on we use $a_{i} = V ^{\max}_{i}$ and $d_{i} = V^{\min}_{i}$ as shorthand. For a particular edge pq shown in (31) we must have each $V(\hat{f}^{i}_{p},\hat{f}^{\pi(i)}_{q}) \leq a_{\pi(i)}$ and so their sum is

(32)

We also know that $V(f^{*}_{p},f^{*}_{q}) \geq d_{\mathrm{lca}(f^{*}_{p},f^{*}_{q})}$ so we can use ratio $\frac{b_{\mathrm{lca}(f^{*}_{p},f^{*}_{q})}}{d_{\mathrm {lca}(f^{*}_{p},f^{*}_{q})}}$ to bound the approximation error at each edge pq appearing in (31), giving upper-bound

(33)

If j is the root of the tree, then $\{p \in\mathcal{A}_{j} \} = \mathcal{P}$ and $\{ pq \in\mathcal{A}_{j} \} = \mathcal{N}$. Using the fact that any ratio $\frac{b_{i}}{d_{i}}$ is bounded from above by quantity c (Definition 7) we arrive at

(34)

(35)

(36)

This completes the proof of Part 1. When there are only smoothness costs, $E(\hat{f}) \leq2c E(f^{*})$ where $\hat{f}$ is the labeling generated at the root of the tree.

Part 2. Derive Coefficient c ₂ for Label Cost Bound

We now revisit from (27) onward but with the assumption that there are hierarchical label costs.

Let E _H (f) denote the total label cost incurred by a labeling f, i.e.the sum of label cost terms. We can bound the label cost $E_{H}(\hat{f}^{j \otimes i})$ of our fused labeling by

$$ E_{H}\bigl(\hat{f}^{j \otimes i}\bigr) \leq E_{H}\bigl(\hat{f}^j\bigr) + \sum_{\substack{L \subseteq \mathcal{L}\setminus\hat{\mathcal{L}}_j\\{L}\cap\hat{\mathcal {L}}_i \neq \emptyset}} H(L) $$

(37)

where $\hat{\mathcal{L}}_{j}$ and $\hat{\mathcal{L}}_{i}$ are the sets of unique labels appearing in $\hat{f}^{j}$ and $\hat{f}^{i}$ respectively.

Recall from Part 1 that, looking at the key inequality (24), we can break the energy terms on each side into parts based on sets $\mathcal{A}_{i}, \overline{\mathcal {A}}_{i}$, and $\partial\mathcal{A}_{i}$. Because $E(\hat{f}^{j \otimes i})|_{\overline{\mathcal{A}}_{i}} = E(\hat{f}^{j})|_{\overline{\mathcal{A}}_{i}}$ these terms cancel out, and we can substitute $E(\hat{f}^{j \otimes i})|_{\mathcal{A}_{i}} = E(\hat{f}^{i})_{\mathcal{A}_{i}}$. Along with bound (37) and canceling the $E_{H}(\hat{f}^{j})$ terms we can now rewrite (24) as

$$ E\bigl(\hat{f}^j\bigr)|_{\mathcal{A}_i \cup \partial\mathcal{A}_i} \leq E\bigl(\hat{f}^i\bigr)|_{\mathcal{A}_i} + E\bigl( \hat{f}^{j \otimes i}\bigr)|_{\partial\mathcal{A}_i} + \sum _{\substack{L \subseteq\mathcal{L}\setminus\hat{\mathcal{L}}_j\\ {L}\cap \hat{\mathcal{L}}_i \neq\emptyset}} H(L). $$

(38)

Again, let $\mathcal{I}^{*} = \{ i \in\mathcal{I}(j) : \mathcal{P}_{i} \neq\emptyset\}$ be the set of child nodes that contain a label used by f ^∗ in their subtree. We sum inequality (38) over all $i \in\mathcal {I}^{*}$ to arrive at a recursive expression, this time incorporating errors incurred by label costs. The key observation is that a particular label cost H(L) appears once on the right-hand side for each element in the set $\mathcal{I}^{*}_{L} = \{ i \in\mathcal{I}^{*} : L \cap\hat{\mathcal{L}} _{i} \neq\emptyset\}$. The sum of inequalities (38) thus implies

(39)

where the quantity in parentheses is identical to that of Part 1.

The above inequality can be recursively expanded for each $E(\hat{f} ^{i})|_{\mathcal{A}_{i}}$ until the recursion stops at a label used by f ^∗. We already know that, after recursive substitution, the quantity in parentheses is bounded by (33). We now must bound the total label cost accumulated by recursive application of (39). The central question is whether a particular subset L that appears in (39) with $|\mathcal{I}^{*}_{L}|>0$ for node j can appear again when we recursively substitute the children $i \in\mathcal{I}^{*}$. If the answer were ‘yes’ then each label cost H(L) could appear more than $|\mathcal{I}^{*}_{L}|$ total times by the end of recursive expansion, leading to a worse bound. Fortunately, Lemma 1 (after this proof) says that this is not the case; each L appearing in the sum for j and child i (38) can never reappear in the sums for i or its children.

From now on we assume j is the root of the tree structure, and so $\hat{f}^{j} = \hat{f}$, i.e.the final labeling output by h-fusion. If we let $\mathcal{H}^{*}$ denote the set of all subsets L generated by recursive substitution of (39), we can thereby write

(40)

Note that the left-hand side of (40) is still $E(\hat{f}^{j})|_{\mathcal{A}_{j}}$ which does not include the label costs incurred by $\hat{f}^{j}$. By adding $E_{H}(\hat{f}^{j})$ to both sides we have $E(\hat{f} ^{j})|_{\mathcal{A}_{j}} + E_{H}(\hat{f}^{j}) = E(\hat{f})$ on the left side, giving a new inequality below.

(41)

All that is left is to re-group the summands in the last three terms (the label cost terms) in a way that proves our theorem. First we rewrite the three sums more explicitly, using $\hat{\mathcal{L}}$ and L ^∗ to denote the unique labels used by $\hat{f}= \hat{f}^{j}$ and f ^∗ respectively.

(42)

First note that if $|\mathcal{I}^{*}_{L}|>1$ then this means $L \supset \mathcal{L}_{i}$ for some $\mathcal{L}_{i} \cap L^{*} \neq\emptyset$ and so L∩L ^∗≠∅ also. We can break the last sum in (42) into two parts based on whether L∩L ^∗≠∅.

(43)

We can also show that $L \in\mathcal{H}^{*} \Rightarrow L \cap \hat{\mathcal{L}}= \emptyset$ as follows. If $L \in\mathcal{H}^{*}$ then there must be some node i such that $L \cap\hat{\mathcal{L}}_{i} = \emptyset$ and $L \subset\mathcal{L}_{i}$. We know from (52) in Lemma 1 that $\hat{\mathcal{L}}\cap\mathcal{L}_{i} \subseteq\hat{\mathcal {L}}_{i}$, so this implies $\emptyset= L \cap\hat{\mathcal{L}}_{i} \supseteq L \cap(\hat {\mathcal{L}}\cap \mathcal{L}_{i}) = L \cap\hat{\mathcal{L}}$. This means the two leftmost sums of (43) have disjoint L and can be bounded by simply $\sum_{L \in\mathcal{H}} H(L)$. It furthermore implies that, for every L appearing in the rightmost sum of (43), the same L must appear in the negative sum. Putting these together we have upper bound on label costs

(44)

We can therefore revise bound (41) to

(45)

Inequality (45) is main result of our theorem.  □

Lemma 1

If label subset L appears in the summand of (38) for node j and child i, then L does not appear in the summands of (38) for any k∈subtree(i).

Proof

To be clear, let us restate the claim more formally. Let $\mathcal{H}^{j \otimes i}$ denote all subsets L appearing in the label cost summands of (38) when applied to node j and child i, i.e.

$$ \mathcal{H}^{j \otimes i} \> \lower1pt \hbox{$\buildrel\mathrm{def} \over= $}\>\{ L : L \cap \hat{\mathcal{L}}_j = \emptyset, L \cap\hat{\mathcal{L}}_i \neq \emptyset\}. $$

(46)

We must prove that $L \in\mathcal{H}^{j \otimes i} \Rightarrow L \notin \mathcal{H}^{k \otimes l}$ for any k∈subtree(i) and $l \in \mathcal{I}(k)$.

First note that for each $L \in\mathcal{H}^{j \otimes i}$ we have

(47)

(48)

By the hierarchical label cost assumption (Definition 4) we can use (47) and (48) to conclude that $L \in \mathcal{H}^{j \otimes i} \Rightarrow L \subset\mathcal{L}_{j}$.

Now consider the set $\mathcal{H}^{j \otimes i} \cap\mathcal{H}^{k \otimes l}$. By the definition (46) an element L of this joint set must satisfy at least the following conditions:

(49)

(50)

(51)

However, no subset L can satisfy all three conditions, as we now show. In the h-fusion algorithm, if $\hat{f}^{i}$ contains a label $\ell\in\mathcal{L}_{k}$, then $\hat{f}^{k}$ must contain ℓ as well—after all, there is no other way that a label in $\mathcal{L}_{k}$ could have propagated up to $\hat{f}^{i}$. This relation can be restated as

$$ \hat{\mathcal{L}}_i \cap\mathcal{L}_k \subseteq\hat{\mathcal{L}}_k \quad\forall k \in\mathrm{subtree}(i). $$

(52)

Starting from (49) we can say

which contradicts requirement (50). Thus $\mathcal{H}^{j \otimes i} \cap\mathcal{H}^{k \otimes l} = \emptyset$ for all k∈subtree(i) and so L cannot reappear. □