Analyzing market baskets by restricted Boltzmann machines

Abstract

We introduce a model which differs from the well-known multivariate logit model (MVL) used to analyze the cross-category dependence in market baskets by the addition of binary hidden variables. This model is called restricted Boltzmann machine (RBM) and new to the marketing literature. Extant applications of the MVL model for higher numbers of categories typically follow a two-step approach as simultaneous maximum likelihood estimation is computationally infeasible. In contrast to the MVL, the RBM can be simultaneously estimated by maximum likelihood even for a higher number of categories as long as the number of hidden variables is moderate. We measure the cross-category dependence by pairwise marginal cross effects which are obtained using estimated coefficients and sampling of baskets. In the empirical study, we analyze market baskets consisting of the 60 most frequently purchased categories of the assortment of a supermarket. For a validation data set, the RBM performs better than the MVL model estimated by maximum pseudo-likelihood. For our data, about 75 % of the baskets are reproduced by the model without cross-category dependence, but 25 % of the baskets cannot be adequately modeled if cross effects are ignored. Moreover, it turns out that both the number of significant cross effects and their relationships can be grasped rather easily.

Appendices

Appendix 1: Maximum likelihood estimation of restricted Boltzmann machines

We apply the BFGS algorithm (see, e.g., Greene 2003) in 50 random restarts and selected the estimated coefficients which lead to the maximum log likelihood value across these restarts for the estimation data.

Initial values $b_j, j=1,\ldots ,J$ are set equal to the ML estimates for the independence model according to expression (10). Coefficients $W_{kj}$ ($k=1, \ldots ,K$ and $j=1,\ldots ,J$) are initialized to random numbers from the normal distribution with mean zero and standard deviation equal to $0.5.$

The first derivative of the log likelihood (9) of a RBM with regard to any coefficient $\theta$ is:

$$\begin{aligned} \frac{\partial \text{ LL}}{\partial \theta } = \sum _ i \frac{\partial \log p^*({\varvec{y}}_i)}{\partial \theta } - I \frac{\partial \log Z_\mathrm{{RBM}}}{\partial \theta } \end{aligned}$$

(19)

In the following, we derive the components of first derivatives of the coefficients. From expression (8) we obtain the unnormalized log probability of a basket:

$$\begin{aligned} \log p^*({\varvec{y}}_i) = {\varvec{b}}^{^{\prime }} {\varvec{y}}_i + \sum _{k=1}^{K} \log \left( 1+ \exp ({\varvec{W}}_{k ,\cdot } {\varvec{y}}_i ) \right) \end{aligned}$$

(20)

The various derivatives of the unnormalized log probability are therefore:

$$\begin{aligned} \frac{\partial \log p^*({\varvec{y}}_i)}{\partial b_j}&= y_{ij} \end{aligned}$$

(21)

$$\begin{aligned} \frac{\partial \log p^*({\varvec{y}}_i)}{\partial W_{kj}}&= 1\,\big /\left(1+\exp \left(-\left( {\varvec{W}}_{k ,\cdot } {\varvec{y}}_i + c_k\right)\right)\right)\!\! y_{ij} \end{aligned}$$

(22)

We investigate $\partial Z_\mathrm{{RBM}} / \partial \theta$ because of

$$\begin{aligned} \frac{\partial \log Z_\mathrm{{RBM}}}{\partial \theta } = \frac{1}{Z_\mathrm{{RBM}}} \frac{\partial Z_\mathrm{{RBM}}}{\partial \theta } \end{aligned}$$

(23)

We write the normalization constant as sum over $2^K$ terms $Z_\mathrm{{RBM}}^{(n)}$:

$$\begin{aligned} Z_\mathrm{{RBM}}&= \sum _{n=1}^{2^K} \prod _{j=1}^{J} \left(1+ \exp ({\varvec{W}}_{\cdot ,j }^{^{\prime }} {\varvec{h}}^{(n)} + b_j) \right) = \sum _{n=1}^{2^K} Z_\mathrm{{RBM}}^{(n)} \end{aligned}$$

(24)

Because of $\partial Z_\mathrm{{RBM}} / \partial \theta = \sum _{n=1}^{2^K} \partial Z_\mathrm{{RBM}}^{(n)} / \partial \theta$ we give the various derivatives $\partial Z_\mathrm{{RBM}}^{(n)} / \partial \theta$ with $h_{k}^{(n)}$ denoting the value that hidden variable $k$ assumes in configuration $n$:

$$\begin{aligned} \frac{\partial Z_\mathrm{{RBM}}^{(n)}}{\partial W_{kj}}&= 1\Big /\left(1+\exp \left(-\left({\varvec{W}}_{\cdot ,j }^{\prime } {\varvec{h}}^{(n)} + b_j\right)\right)\right) \, Z_\mathrm{{RBM}}^{(n)} h_{k}^{(n)}\end{aligned}$$

(25)

$$\begin{aligned} \frac{\partial Z_\mathrm{{RBM}}^{(n)}}{\partial b_{j}}&= 1\Big /\left(1+\exp \left(-\left({\varvec{W}}_{\cdot ,j }^{\prime } {\varvec{h}}^{(n)} + b_j\right)\right)\right) \, Z_\mathrm{{RBM}}^{(n)} \end{aligned}$$

(26)

Appendix 2: Maximum pseudo-likelihood estimation of the multivariate logit model

We estimate the MVL model by means of the BFGS algorithm (see, e.g., Greene 2003) using first derivatives of the log pseudo-likelihood. After rewriting and simplifying we obtain the following expressions for first derivatives of the log pseudo-likelihood of any basket $i$ with respect to constants $a_j$ and cross-category coefficients $V_{jl}$ from expressions (12) and (2):

$$\begin{aligned} \frac{\partial \text{ LPL}_i }{\partial a_j}&= y_{ij} - p(y_{ij} | {\varvec{y}}_{i,-j}) \quad \text{ for} \ j=1, \ldots , J \end{aligned}$$

(27)

$$\begin{aligned} \frac{\partial \text{ LPL}_i }{\partial V_{jl}}&= 2 y_{ij} y_{il} - y_{il} p(y_{ij} | {\varvec{y}}_{i,-j}) - y_{ij} p(y_{il} | {\varvec{y}}_{i,-l}) \quad \text{ for} \ l>j \end{aligned}$$

(28)