Divide-and-Conquer Parallelism for Learning Mixture Models

Abstract

From the viewpoint of load balancing among processors, the acceleration of machine-learning algorithms by using parallel loops is not realistic for some models involving hierarchical parameter estimation. There are also other serious issues such as memory access speed and race conditions. Some approaches to the race condition problem, such as mutual exclusion and atomic operations, degrade the memory access performance. Another issue is that the first-in-first-out (FIFO) scheduler supported by frameworks such as Hadoop can waste considerable time on queuing and this will also affect the learning speed. In this paper, we propose a recursive divide-and-conquer-based parallelization method for high-speed machine learning. Our approach exploits a tree structure for recursive tasks, which enables effective load balancing. Race conditions are also avoided, without slowing down the memory access, by separating the variables for summation. We have applied our approach to tasks that involve learning mixture models. Our experimental results show scalability superior to FIFO scheduling with an atomic-based solution to race conditions and robustness against load imbalance.

A General EM Algorithm

1.1 A.1 EM on GMM

The GMM is a popular probabilistic model described by a weighted linear sum of K normal distributions:

$$\begin{aligned} p(\varvec{x}) = \sum _{k=1}^K w_k \mathcal {N}(\varvec{x};\varvec{\mu }_k,S_k), \end{aligned}$$

(1)

where $w_k$ is the weight, $\varvec{\mu }_k$ is the mean, and $S_k$ is the covariance matrix of the kth normal distribution. An observation item $\varvec{x}$ is generated by a normal distribution selected with a probability of $w_k$. We transcribe parameters $\theta _k = (w_k, \varvec{\mu }_k, S_k)$ for the sake of simplicity, and $\theta$ is the set of all $\theta _k$. The likelihood function $\mathcal {L}(\theta )$ indicates how likely it is that the probabilistic model regenerates the training dataset. Assuming independence among observation items, $\mathcal {L}(\theta )$ is equal to the joint probability of all observation data. $\mathcal {L}$ is defined in log-likelihood terms because $p(\varvec{x}_n|\theta )$ is very small:

$$\begin{aligned} \mathcal {L}(\theta ) = \sum _n^N \log \sum _k^K w_k \mathcal {N}(\varvec{x}_n;\varvec{\mu }_k,S_k). \end{aligned}$$

(2)

In the EM context, we need only maximize $\mathcal {L}$. However, because a GMM is a latent-variable model, it requires step-by-step improvement. The posterior probability $q_{nk}$ that the nth observation item $\varvec{x}_n$ is generated by the kth normal distribution is:

$$\begin{aligned} q_{nk} = \frac{w_k \mathcal {N}(\varvec{x}_n;\varvec{\mu }_k,S_k)}{\displaystyle \sum _k^K w_k \mathcal {N}(\varvec{x}_n;\varvec{\mu }_k,S_k)}. \end{aligned}$$

(3)

Of the two repeated steps, the E-step calculates $q_{nk}$ for all pairs of data $\varvec{x}_n$ and the kth normal distribution, and the M-step updates the parameters as follows:

$$\begin{aligned} \hat{w}_k&= \frac{1}{N} \sum _n^N q_{nk}, \end{aligned}$$

(4)

$$\begin{aligned} \hat{\varvec{\mu }}_k&= \frac{1}{N\hat{w}_k} \sum _n^N q_{nk} \varvec{x}_n, \end{aligned}$$

(5)

$$\begin{aligned} \hat{S}_k&= \frac{1}{N\hat{w}_k} \sum _n^N q_{nk} (\varvec{x}_n - \hat{\varvec{\mu }}_k)^T(\varvec{x}_n - \hat{\varvec{\mu }}_k). \end{aligned}$$

(6)

The E-step and M-step are repeated alternately until $\mathcal {L}$ converges. In practice, the covariance matrix $S_k$ is assumed to be a diagonal matrix and the calculation is therefore simplified as follows:

$$\begin{aligned} \hat{S}_{kd} = \frac{1}{N\hat{w}_k} \left( \sum _n^N \hat{q}_{nk} \varvec{x}_{nd}^2\right) - \hat{\varvec{\mu }}_{kd}^2. \end{aligned}$$

(7)

In the E-step, $N \times K$ $q_{nk}$ is calculated, and in the M-step, $q_{nk}$ is summed in the N axis and the parameter $\theta _k$ is updated. However, the posterior table can be too large and can exceed the hard-disk capacity when N is very large. Because of poor memory throughput, the processing speed will then degrade greatly. To avoid this condition, the parallel EM algorithm requires a large memory space.

1.2 A.2 EM on HPMM

Kinoshita et al. used an HPMM to detect traffic incidents [10]. They assumed that probe-car records follow a hierarchical PMM and that each road segment has its own local parameters. In their model, the probability of a single record $\varvec{x}$ in a segment s is described as follows:

$$\begin{aligned} p(\varvec{x}|s) = \sum _{k=1}^K w_{sk} \mathcal {P}(\varvec{x};\varvec{\mu }_k), \end{aligned}$$

(8)

where $w_{sk}$ is the kth Poisson distribution’s weight in segment s, and $\varvec{\mu }_k$ is the kth Poisson distribution’s mean. $w_{sk}$ is particular to the segment, whereas $\varvec{\mu }_k$ is common to all segments. The log-likelihood $\mathcal {L}(\theta )$ is defined as follows:

$$\begin{aligned} \mathcal {L}(\theta ) = \sum _{s=1}^S \sum _{n=1}^{N_s} \log \sum _{k=1}^K w_{sk} \mathcal {P}(\varvec{x}_{sn};\varvec{\mu }_k), \end{aligned}$$

(9)

where $N_s$ is the number of records in segment s. As for GMMs, we must calculate the posterior probability $q_{snk}$ that the nth record $\varvec{x}_{sn}$ in segment s is generated by the kth Poisson distribution for all pairs of (s, n, k) in each E-step:

$$\begin{aligned} q_{snk} = \frac{w_{sk} \mathcal {P}(\varvec{x}_{sn};\varvec{\mu }_k)}{\displaystyle \sum _{k=1}^K w_{sk} \mathcal {P}(\varvec{x}_{sn};\varvec{\mu }_k)}. \end{aligned}$$

(10)

In the M-step, the weight $w_{sk}$ and mean $\varvec{\mu }_k$ are recalculated:

$$\begin{aligned} \hat{w}_{sk}&= \frac{1}{N_s} \sum _{n=1}^{N_s} q_{snk}, \end{aligned}$$

(11)

$$\begin{aligned} \hat{\varvec{\mu }}_k&= \frac{\displaystyle \sum _{s=1}^S \sum _{n=1}^{N_s} q_{snk} \varvec{x}_{sn}}{\displaystyle \sum _{s=1}^S \sum _{n=1}^{N_s} q_{snk}}. \end{aligned}$$

(12)

Each road segment has a massive number of records, with the actual number varying greatly from segment to segment. This implies that we should take measures against load imbalance.