Calibrating AdaBoost for phoneme classification

Abstract

Phoneme classification is a classification sub-task of automatic speech recognition (ASR), which is essential in order to achieve good speech recognition accuracy. However, unlike most classification tasks, besides finding the correct class, providing good posterior scores is also an important requirement of it. Partly because of this, formerly Gaussian Mixture Models, while recently Artificial Neural Networks (ANNs) are used in this task, while other common machine learning methods like Support Vector Machines and AdaBoost.MH are applied only rarely. In a previous study, we showed that AdaBoost.MH can match the performance of ANNs in terms of classification accuracy, but lags behind it when utilizing its output in the speech recognition process. This is in part due to the imprecise posterior scores that AdaBoost.MH produces, which is a well-known weakness of this method. To improve the quality of posterior scores produced, it is common to perform some kind of posterior calibration. In this study, we test several posterior calibration techniques in order to improve the overall performance of AdaBoost.MH. We found that posterior calibration is a good way to improve ASR accuracy, especially when we integrate the speech recognition process into the calibration workflow.

A Proof of Proposition 1

Proof

Let us assume a strong classifier $\mathbf{f}^{(T)} (\mathbf{x})$. Then, we seek to find weak classifier $\mathbf{h}(.)$ such that $J( \mathbf{f}^{(T)} + c \mathbf{h})$ is minimized. By using the linearity of the expectation, one may consider the second-order expansion of $J ( \mathbf{f}^{(T)} + c \mathbf{h})$ for fixed $c \in \mathbb {R}_{+}$ and $\mathbf{h}( x ) = (0, \dots , 0)$ as

$$\begin{aligned}&J ( \mathbf{f}^{(T)} + c \mathbf{h}) \\&\quad = \mathbb {E}\left[ \sum _{ \ell = 1 }^K I ( \mathbf{y}, \ell ) \exp (-\,y_{\ell } ( f_{\ell }^{(T)} (\mathbf{x}) + c h_{\ell } (\mathbf{x})))\right] \\&\quad = \sum _{ \ell = 1 }^K \mathbb {E}\left[ I ( \mathbf{y}, \ell ) \exp (-\,y_{\ell } ( f_{\ell }^{(T)} (\mathbf{x}) + c h_{\ell } (\mathbf{x})))\right] \\&\quad \approx \sum _{ \ell = 1 }^K \mathbb {E}\left[ I ( \mathbf{y}, \ell ) \exp (-\,y_{\ell } f_{\ell }^{(T)} (\mathbf{x}) ) \right. \\&\qquad \quad \left. ( 1 - c y_{\ell } h_{\ell }(\mathbf{x}) + c^2_{\ell } y_{\ell }^2 h_{\ell }(\mathbf{x})^2/2 ) \right] \\&\quad = \sum _{ \ell = 1 }^K \mathbb {E}\left[ I ( \mathbf{y}, \ell ) \exp (-\,y_{\ell } f_{\ell }^{(T)} (\mathbf{x}) ) \right. \\&\quad \left. ( 1 - c y_{\ell } h_{\ell }(\mathbf{x}) + c^2_{\ell }/2 ) \right] \end{aligned}$$

where the last equation holds because $h_{\ell } (\mathbf{x})\in \{-\,1, 1 \}$ and $y_{\ell }\in \{-\,1, 1 \}$. This means that minimizing the multi-class exponential loss with respect to $\mathbf{h}$ is equivalent to maximizing the weighted expectation

$$\begin{aligned} \mathbb {E}\left[ \sum _{\ell =1}^K w(\mathbf{x}, \mathbf{y}, \ell )y_{\ell } h_{\ell } (\mathbf{x}) \right] \end{aligned}$$

(31)

where

$$\begin{aligned} w( \mathbf{x}, \mathbf{y}, \ell ) = I( \mathbf{y}, \ell ) \exp \left( -\,y_{\ell } f_{\ell }^{(T)} (\mathbf{x})\right) . \end{aligned}$$

Note that the weak learner maximizes the edge given in (14) computed on a finite dataset, which is an estimate of the weighted expectation.

The label distribution in the multi-class case for fixed $\mathbf{x}$ is a multinomial distribution with $(p_1, \dots ,p_K ) \in \varDelta _{K}$, where $\varDelta _{K}$ is the K dimensional probability simplex. Note that $(p_1, \dots ,p_K )$ depends on $\mathbf{x}$, but with a slight abuse of notation we shall not indicate this dependence if $\mathbf{x}$ is fixed and this does not lead to any confusion. We shall write $(p_1 (\mathbf{x}), \dots ,p_K (\mathbf{x}))$ to denote the dependence on $\mathbf{x}$.

Recall that the goal is to maximize the weighted expectation given in (31). As a next step we are going to compute the form of the optimal weak classifier. Let us define the vector $\mathbf {e}_{\ell }$ as

$$\begin{aligned} e_{\ell ,\ell ^{\prime } } = {\left\{ \begin{array}{ll} 1, &{} \ell = \ell ^{\prime } \\ -\,1, &{} \text {otherwise} \end{array}\right. } \end{aligned}$$

Furthermore, assume that we are given a weak classifier $\mathbf{h}(. )$ such that $h_{\widehat{\ell }}( \mathbf{x}) = 1$ for some $\widehat{\ell }$ and $h_{\ell }( \mathbf{x}) = 0$ otherwise. Then, the numerator of (31) can be written for a fixed $\mathbf{x}$ with a label distribution $(p_1, \dots ,p_K )$ as

$$\begin{aligned} \mathbb {E}&\left[ \sum _{\ell =1}^K w(\mathbf{x}, \mathbf{y}, \ell ) y_{\ell } h_{\ell } (\mathbf{x}) \vert \mathbf{x}\right] \nonumber \\&= \sum _{\ell = 1}^K p_{\ell } \left[ w(\mathbf{x}, \mathbf {e}_{\ell }, \ell ) h_{\ell } (\mathbf{x}) - \sum _{\ell ^{\prime } \ne \ell } w(\mathbf{x}, \mathbf {e}_{\ell }, \ell ^{\prime } ) h_{\ell ^{\prime }} (\mathbf{x}) \right] \nonumber \\&= p_{\widehat{\ell }} \left[ w(\mathbf{x}, \mathbf {e}_{\widehat{\ell }}, \widehat{\ell } ) + \sum _{\ell ^{\prime } \ne \widehat{\ell } } w(\mathbf{x}, \mathbf {e}_{\widehat{\ell }}, \ell ^{\prime } ) \right] \nonumber \\&\quad + \sum _{\ell \ne \widehat{\ell } } p_{\ell } \left[ -\, w(\mathbf{x}, \mathbf {e}_{\ell }, \ell ) - w(\mathbf{x}, \mathbf {e}_{\ell }, \widehat{\ell } ) + \sum _{\ell ^{\prime } \ne \ell , \widehat{\ell } } w(\mathbf{x}, \mathbf {e}_{\ell }, \ell ^{\prime } ) \right] \nonumber \\&= p_{\widehat{\ell }} \sum _{\ell =1 }^K w(\mathbf{x}, \mathbf {e}_{\widehat{\ell }}, \ell ) \nonumber \\&\quad + \sum _{\ell \ne \widehat{\ell } } p_{\ell } \left[ -\, 2w(\mathbf{x}, \mathbf {e}_{\ell }, \ell ) - 2w(\mathbf{x}, \mathbf {e}_{\ell }, \widehat{\ell } ) + \sum _{\ell ^{\prime } = 1 }^K w(\mathbf{x}, \mathbf {e}_{\ell }, \ell ^{\prime } ) \right] \nonumber \\&= \sum _{\ell = 1 }^K \sum _{\ell ^{\prime } = 1 }^K w(\mathbf{x}, \mathbf {e}_{\ell }, \ell ^{\prime } ) -2 \sum _{\ell \ne \widehat{\ell } } p_{\ell } \left[ w(\mathbf{x}, \mathbf {e}_{\ell }, \ell ) + w(\mathbf{x}, \mathbf {e}_{\ell }, \widehat{\ell } ) \right] \nonumber \\&= \sum _{\ell = 1 }^K \sum _{\ell ^{\prime } = 1 }^K w(\mathbf{x}, \mathbf {e}_{\ell }, \ell ^{\prime } ) - 2 \sum _{\ell =1 }^K p_{\ell } w(\mathbf{x}, \mathbf {e}_{\ell }, \ell ) \nonumber \\&\quad + 2 \left[ p_{\widehat{\ell }} w(\mathbf{x}, \mathbf {e}_{\widehat{\ell }}, \widehat{\ell } ) - \sum _{\ell \ne \widehat{\ell } } p_{\ell } w(\mathbf{x}, \mathbf {e}_{\ell }, \widehat{\ell } ) \right] \end{aligned}$$

(32)

Since only the last term depends on $\widehat{\ell }$, the optimal classifier can be written in the form

$$\begin{aligned}&\widehat{\ell }(\mathbf{x}) = \mathrm{arg\, max}_{1\le \ell \le K } \\&\quad \left[ p_{\ell }(\mathbf{x}) w(\mathbf{x}, \mathbf {e}_{\ell }, \ell ) - \sum _{\ell ^{\prime } \ne \ell } p_{\ell ^{\prime }}(\mathbf{x}) w(\mathbf{x}, \mathbf {e}_{\ell ^{\prime }}, \ell )\right] \\&= \mathrm{arg\, max}_{1\le \ell \le K } \left[ p_{\ell }(\mathbf{x}) w(\mathbf{x}, \mathbf {e}_{\ell }, \ell ) \right] , \end{aligned}$$

and hence, the optimal classifier $\mathbf{h}( \mathbf{x})$ can be written in the form of

$$\begin{aligned} h_{\ell } (\mathbf{x}) = {\left\{ \begin{array}{ll} 1 ,&{} \text {if }~ \ell = \widehat{\ell }(\mathbf{x})\\ -\,1, &{} \text {otherwise} \end{array}\right. } . \end{aligned}$$

(33)

Considering (32) over the joint distribution of the labels and features, we get that the classifier which is in the form of (33) minimizes the population-level edge defined as

$$\begin{aligned} \gamma ^* = \mathbb {E}\left[ \sum _{\ell =1}^K w(\mathbf{x}, \mathbf{y}, \ell ) y_{\ell } h_{\ell } (\mathbf{x}) \right] , \end{aligned}$$

where the expectation is taken over the joint distribution.

As a next step, one can determine the optimal coefficient c of $\mathbf{h}(.)$ given in (33) by optimizing $J ( \mathbf{f}^{(T)} + c \mathbf{h})$:

$$\begin{aligned} c&= \mathop {\text {argmin}}\limits _{0< c} \mathbb {E}\left[ \sum _{\ell =1}^K w(\mathbf{x}, \mathbf{y}, \ell ) \exp (-\,c y_{\ell } h_{\ell } (\mathbf{x}) )\right] \\&\le \mathop {\text {argmin}}\limits _{0 < c} \mathbb {E}\left[ \sum _{\ell =1}^K w(\mathbf{x}, \mathbf{y}, \ell ) \left( \frac{1+ y_{\ell } h_{\ell } (\mathbf{x})}{2} \exp (-\,c) \right. \right. \\&\quad \left. \left. + \frac{1- y_{\ell } h_{\ell } (\mathbf{x})}{2} \exp (c) \right) \right] \end{aligned}$$

Then, one can compute analytically that the right-hand side is minimized by

$$\begin{aligned} c = \frac{1}{2}\log \frac{1+\gamma ^*}{1-\gamma ^*}, \end{aligned}$$

where $\gamma ^*$ is the population-level edge. Note that the AdaBoost.MH computes the coefficient of the weak classifier in a similar way by using the edge on a finite training data instead of taking the population-level edge. The rest of the proof regarding the weight update is analogous to that of Theorem 1 in Friedman et al. (2000). $\square$