An Extension of Iterative Scaling for Decision and Data Aggregation in Ensemble Classification

Abstract

Improved iterative scaling (IIS) is an algorithm for learning maximum entropy (ME) joint and conditional probability models, consistent with specified constraints, that has found great utility in natural language processing and related applications. In most IIS work on classification, discrete-valued “feature functions” are considered, depending on the data observations and class label, with constraints measured based on frequency counts, taken over hard (0–1) training set instances. Here, we consider the case where the training (and test) set consist of instances of probability mass functions on the features, rather than hard feature values. IIS extends in a natural way for this case. This has applications (1) to ME classification on mixed discrete-continuous feature spaces and (2) to ME aggregation of soft classifier decisions in ensemble classification. Moreover, we combine these methods, yielding a method, with proved learning convergence, that jointly performs (soft) decision-level and feature-level fusion in making ensemble decisions. We demonstrate favorable comparisons against standard Adaboost.M1, input-dependent boosting, and other supervised combining methods, on data sets from the UC Irvine Machine Learning repository.

Appendices

Appendix A

For an arbitrary parameter vector $\underline{\gamma}$, the conditional log-likelihood is

$$L{\left( {\underline{\gamma } } \right)} = {\sum\limits_{t = 1}^T {\ln } }{\left[ {\frac{{e^{{{\sum\limits_{i = 1}^{N_{d} } {{\sum\limits_{j \in \mathcal{A}_{i} } {P{\left[ {F_{i} = j\left| t \right.} \right]}\gamma {\left( {C = c^{{{\left( t \right)}}} ,F_{i} = j} \right)}} }} }}} }}{{{\sum\limits_{k = 1}^{N_{c} } {e^{{{\sum\limits_{i = 1}^{N_{d} } {{\sum\limits_{j \in \mathcal{A}_{i} } {P{\left[ {F_{i} = j\left| t \right.} \right]}\gamma {\left( {C = k,F_{i} = j} \right)}} }} }}} } }}}} \right]}$$

(22)

For a change in the parameter vector $\underline{\Delta\gamma}$, the change in log-likelihood is $L(\underline{\gamma}+\underline{\Delta\gamma})-L(\underline{\gamma})$. Using the identity $-\ln(\alpha)\geq 1-\alpha$, we obtain the lower bound

$$\begin{array}{*{20}l} {{L{\left( {\underline{\gamma } + \underline{{\Delta \gamma }} } \right)} - L{\left( {\underline{\gamma } } \right)} \geqslant {\sum\limits_{t = 1}^T {{\left[ {{\sum\limits_{i = 1}^{N_{d} } {{\sum\limits_{j \in \mathcal{A}_{i} } {P{\left[ {F_{i} = j\left| t \right.} \right]}\Delta \gamma {\left( {C = c^{{{\left( t \right)}}} ,F_{i} = j} \right)} + 1} }} }} \right]}} }} \hfill} \\ {{ - {\sum\limits_{t = 1}^T {{\sum\limits_{k = 1}^{N_{c} } {P{\left[ {C = k\left| {P_{{\underline{f} ^{{{\left( t \right)}}} }} } \right.} \right]} \times e^{{{\sum\limits_{i = 1}^{N_{d} } {{\sum\limits_{j \in \mathcal{A}_{i} } {P{\left[ {F_{i} = j\left| t \right.} \right]}} }} }\Delta \gamma {\left( {C = k,F_{i} = j} \right)}}} \widehat{ = }B{\left( {\underline{{\Delta \gamma }} \left| {\underline{\gamma } } \right.} \right)}} }} }} \hfill} \\ \end{array} $$

With $\underline{\Delta\gamma}$ chosen so that $B(\underline{\Delta\gamma}|\underline{\gamma}) \geq 0$ there is improvement in log-likelihood. The obvious approach then is to maximize $B(\underline{\Delta\gamma}|\underline{\gamma})$. However, there is coupled dependence in $B(\underline{\Delta\gamma}|\underline{\gamma})$ on the individual components $\left\{\Delta\gamma(C=c,{F}_{i}=j)\right\}$, which would necessitate a complicated joint optimization. Thus, we seek an auxiliary function that decouples this dependence. We first rewrite

$$\begin{array}{*{20}l} {{B{\left( {\underline{{\Delta \gamma }} + \underline{\gamma } } \right)} = {\sum\limits_{t = 1}^T {{\left[ {{\sum\limits_{i = 1}^{N_{d} } {{\sum\limits_{j \in \mathcal{A}_{i} } {P{\left[ {F_{i} = j\left| t \right.} \right]}\Delta \gamma {\left( {C = c^{{{\left( t \right)}}} ,F_{i} = j} \right)} + 1} }} }} \right]}} }} \hfill} \\ {{ - {\sum\limits_{t = 1}^T {{\sum\limits_{k = 1}^{N_{c} } {P{\left[ {C = k\left| {P_{{\underline{f} ^{{{\left( t \right)}}} }} } \right.} \right]} \times e^{{{\sum\limits_{i = 1}^{N_{d} } {{\sum\limits_{j \in \mathcal{A}_{i} } {\frac{{P{\left[ {F_{i} = j\left| t \right.} \right]}}}{{N_{d} }}} }} }{\left[ {N_{d} \Delta \gamma } \right]}{\left( {C = k,F_{i} = j} \right)}}} } }} }} \hfill} \\ \end{array} $$

(23)

Then, we note that $\sum\nolimits_{i=1}^{N_{d}}\sum\nolimits_{j\in{\cal A}_{i}}\frac{P[{F}_{i}=j|t]}{N_{d}}=1$ i.e., $\left\{\frac{1}{N_d}P[{F}_{i}=j|t], \hspace{0.05in} i=1,...,N_d,\hspace{0.05in} j\in{\cal A}_{i}\right\}$ is an instance of the joint pmf $P[I,F_I]$, associated with first selecting a feature $i \in \left\{1,...,N_d\right\}$ and then a feature value $f_i \in {\cal A}_{i}$. Applying Jensen’s inequality $e^{\sum_{x} p(x)q(x)} \leq \sum_{x} p(x) e^{q(x)}$ to the right hand side of Eq. (23), we have

$$\begin{array}{*{20}l} {{B{\left( {\underline{{\Delta \gamma }} + \underline{\gamma } } \right)} \geqslant T + {\sum\limits_{t = 1}^T {{\sum\limits_{i = 1}^{N_{d} } {{\sum\limits_{j \in \mathcal{A}_{i} } {P{\left[ {F_{i} = j\left| t \right.} \right]}\Delta \gamma {\left( {C = c^{{{\left( t \right)}}} ,F_{i} = j} \right)}} }} }} }} \hfill} \\ {{ - \frac{1}{{N_{d} }}{\sum\limits_{t = 1}^T {{\sum\limits_{k = 1}^{N_{c} } {{\sum\limits_{i = 1}^{N_{d} } {{\sum\limits_{j \in \mathcal{A}_{i} } {P{\left[ {C = k\left| {P_{{\underline{f} ^{{{\left( t \right)}}} }} } \right.} \right]}P{\left[ {F_{i} = j\left| t \right.} \right]}e^{{N_{d} \Delta \gamma {\left( {C = k,F_{i} = j} \right)}}} } }\widehat{ = }A{\left( {\underline{{\Delta \gamma }} \left| {\underline{\gamma } } \right.} \right)}} }} }} }.} \hfill} \\ \end{array} $$

(24)

Thus $L(\underline{\gamma}+\underline{\Delta\gamma})-L(\underline{\gamma}) \geq B(\underline{\Delta\gamma}|\underline{\gamma}) \geq A(\underline{\Delta\gamma}|\underline{\gamma})$, i.e., we have a new, not as tight, lower bound. Let $\underline{\Delta \gamma^{\ast}} = \arg\max_{\underline{\Delta \gamma}} A(\cdot)$. Since it is easy to verify that $A(\underline{0}|\underline{\gamma})=0$, it must be true that $A(\underline{\Delta \gamma^{\ast}}|\underline{\gamma}) \geq 0$, i.e., property A1 in Section 2.3 is satisfied by this function. Moreover, $A(\underline{\Delta\gamma}|\underline{\gamma})$ can be additively decoupled into individual terms each depending on a single $\Delta\gamma(C=k,{F}_{n}=q)$. Differentiating $A(\underline{\Delta\gamma}|\underline{\gamma})$ with respect to $\Delta\gamma(C=k,{F}_{n}=q)$ and equating to 0 we get the choice of $\underline{\Delta\gamma}$ to maximize $A(\underline{\Delta\gamma}|\underline{\gamma})$, i.e.,

$$\begin{array}{*{20}l} {{\Delta \gamma ^{ * } {\left( {C = k,F_{n} = q} \right)} = \frac{1}{{N_{d} }}\ln {\left[ {\frac{{{\sum\limits_{t = 1;C = k}^T {P{\left[ {F_{n} = q\left| t \right.} \right]}} }}}{{{\sum\limits_{t = 1}^T {P{\left[ {F_{n} = q\left| t \right.} \right]}P{\left[ {C = k\left| {P_{{\underline{f} ^{{{\left( t \right)}}} }} } \right.} \right]}} }}}} \right]}} \hfill} \\ {{ = \frac{1}{{N_{d} }}\ln \frac{{P_{g} {\left[ {C = k,F_{n} = q} \right]}}}{{P_{m} {\left[ {C = k,F_{n} = q} \right]}}},\forall k,n,q.} \hfill} \\ \end{array} $$

(25)

Appendix B

Theorem 1

Consider the function $A(\underline{\Delta\gamma}|\underline{\gamma})$ defined in Eq. (24). Let $\underline{\Delta \gamma^{\ast}} = \arg\max_{\underline{\Delta \gamma}} A(\underline{\Delta \gamma}|\underline{\gamma})$. Then, $A(\underline{\Delta\gamma^{\ast}}|\underline{\gamma})=0$ $ff\;\underline{{\Delta \gamma ^{ * } }} = \underline{0}$. When this occurs the constraints (4) are all met.

Proof

Setting $\dfrac{\partial A(\underline{\Delta\gamma}|\underline{\gamma})}{\partial\Delta\gamma(F_{i}=j,C=c)}=0$ gives the solution in Eq. (25). Moreover, this is a maximum since $\frac{{\partial ^{2} A{\left( {\underline{{\Delta \gamma }} \left| {\underline{\gamma } } \right.} \right)}}}{{\partial \Delta \gamma ^{2} {\left( {F_{i} = j,C = c} \right)}}} < 0\forall i,j,c$. Thus, $\underline{\Delta \gamma^{\ast}} = \left\{ \Delta\gamma^{\ast}(F_{i}=j,C=c) \hspace{0.05in} \forall i,\hspace{0.05in} \forall j \in {\cal A}_i, \hspace{0.05in} c \in C \right\}.$

Plugging the solution for $\underline{\Delta \gamma^{\ast}}$ back into $A(\underline{\Delta\gamma}|\underline{\gamma})$ in Eq. (24) and simplifying gives

$$\begin{array}{*{20}l} {{A{\left( {\underline{{\Delta \gamma ^{ * } }} \left| {\underline{\gamma } } \right.} \right)} = T + {\sum\limits_t^T {{\sum\limits_i {{\sum\limits_j {{\sum\limits_c {P{\left[ {F_{i} = j\left| t \right.} \right]}} }} }} }} }} \hfill} \\ {{\frac{1}{{N_{d} }}\ln {\left[ {\frac{{{\sum\limits_{t = 1:c^{{{\left( t \right)}}} = c}^T {P{\left[ {F_{i} = j\left| t \right.} \right]}} }}}{{{\sum\nolimits_{t = 1}^T {P{\left[ {F_{i} = j\left| t \right.} \right]}P{\left[ {C = c\left| {P_{{\underline{f} ^{{{\left( t \right)}}} }} } \right.} \right]}} }}}} \right]}} \hfill} \\ {{ - \frac{1}{{N_{d} }}{\sum\limits_i {{\sum\limits_j {{\sum\limits_{c = 1}^C {{\sum\limits_{t = 1:c^{{{\left( t \right)}}} = c} {P{\left[ {F_{i} = j\left| t \right.} \right]}} }} }} }} }} \hfill} \\ \end{array} $$

(26)

Noting that the third term equals T, we have

$$\begin{array}{*{20}l} {{A{\left( {\underline{{\Delta \gamma ^{ * } }} \left| {\underline{\gamma } } \right.} \right)} = \frac{1}{{N_{d} }}{\sum\limits_t^T {{\sum\limits_i {{\sum\limits_j {{\sum\limits_c {P{\left[ {F_{i} = jt} \right]}} }} }} }} }\ln {\left[ {\frac{{{\sum\limits_{t = 1:c^{{{\left( t \right)}}} = c}^T {P{\left[ {F_{i} = j\left| t \right.} \right]}} }}}{{{\sum\limits_{t = 1}^T {P{\left[ {F_{i} = j\left| t \right.} \right]}P{\left[ {C = c\left| {P_{{\underline{f} ^{{{\left( t \right)}}} }} } \right.} \right]}} }}}} \right]}} \hfill} \\ {{ = - \frac{1}{{N_{d} }}{\sum\limits_t^T {{\sum\limits_i {{\sum\limits_j {{\sum\limits_c {P{\left[ {F_{i} = j\left| t \right.} \right]}\ln {\left[ {\frac{{\frac{1}{T}{\sum\limits_{t = 1}^T {P{\left[ {F_{i} = j\left| t \right.} \right]}P{\left[ {C = c\left| {P_{{\underline{f} ^{{{\left( t \right)}}} }} } \right.} \right]}} }}}{{\frac{1}{T}{\sum\limits_{t = 1:c^{{{\left( t \right)}}} = c}^T {P{\left[ {F_{i} = j\left| t \right.} \right]}} }}}} \right]}} }} }} }} }} \hfill} \\ \end{array} $$

Now, since $E(-\ln(\cdot)) \ge -\ln(E(\cdot))$ by Jensen’s inequality, we have

$$A{\left( {\underline{{\Delta \gamma ^{ * } }} \left| {\underline{\gamma } } \right.} \right)} \geqslant - \frac{T}{{N_{d} }}{\sum\limits_i {\ln } }{\left[ {{\sum\limits_{j,c} {\frac{1}{T}{\sum\limits_{t = 1:c^{{{\left( t \right)}}} = c}^T {P{\left[ {F_{i} = j\left| t \right.} \right]}{\left[ {\frac{{\frac{1}{T}{\sum\limits_{t\prime = 1}^T {P{\left[ {F_{i} = j\left| {t\prime } \right.} \right]}P{\left[ {C = c\left| {P_{{\underline{f} ^{{{\left( {t\prime } \right)}}} }} } \right.} \right]}} }}}{{\frac{1}{T}{\sum\limits_{t\prime = 1:c^{{t\prime }} = c}^T {P{\left[ {F_{i} = j\left| {t\prime } \right.} \right]}} }}}} \right]}} }} }} \right]}$$

$$\begin{array}{*{20}c} { = - \frac{T}{{N_{d} }}{\sum\limits_i {\ln } }{\left[ {{\sum\limits_j {{\sum\limits_c {\frac{1}{T}{\sum\limits_{t\prime = 1}^T {P{\left[ {F_{i} = j\left| {t\prime } \right.} \right]}P{\left[ {C = c\left| {P_{{\underline{f} ^{{{\left( {t\prime } \right)}}} }} } \right.} \right]}} }} }} }} \right]}} \\ { = - \frac{T}{{N_{d} }}{\sum\limits_i {\ln {\left( 1 \right)} = 0} }} \\ \end{array} $$

i.e., $A(\underline{\Delta \gamma^{\ast}}|\underline{\gamma}) \ge 0$. Finally by strict convexity of $\ln\left(\cdot\right)$, equality is achieved iff the argument of the $\ln\left(\cdot\right)$ is 1. But this occurs iff

$${\sum\limits_{t = 1}^T {P{\left[ {F_{i} = j\left| t \right.} \right]}P{\left[ {C = c\left| {P_{{\underline{f} ^{{{\left( t \right)}}} }} } \right.} \right]}} } = {\sum\limits_{t = 1;C = c}^T {P{\left[ {F_{i} = j\left| t \right.} \right]}\forall i,j,c} }$$

i.e., by Eq. (25), iff $\underline{\Delta \gamma^{\ast}} = \underline{0}$. Clearly, when this occurs, the constraints are met.▪