Embedded heterogeneous feature selection for conjoint analysis: A SVM approach using L1 penalty

Abstract

This paper presents a novel embedded feature selection approach for Support Vector Machines (SVM) in a choice-based conjoint context. We extend the L1-SVM formulation and adapt the RFE-SVM algorithm to conjoint analysis to encourage sparsity in consumer preferences. This sparsity can be attributed to consumers being selective about the attributes they consider when evaluating alternatives in choice tasks. Given limited individual data in choice-based conjoint, we control for heterogeneity by pooling information across consumers and shrinking the individual weights of the relevant attributes towards a population mean. We tested our approach through an extensive simulation study that shows that the proposed approach can capture the sparseness implied by irrelevant attributes. We also illustrate the characteristics and use of our approach on two real-world choice-based conjoint data sets. The results show that the proposed method has better predictive accuracy than competitive approaches, and that it provides additional information at an individual level. Implications for product design decisions are discussed.

Appendices

Appendix A: HB mixed logit estimation

1.1 Prior and full conditional distributions

We denote by 𝜃 _i the set of random-effect parameters.

1.2 Priors

Random-effect parameters 𝜃 _i

$$\begin{array}{@{}rcl@{}} \boldsymbol{\theta}_{i} &\sim& N(\boldsymbol{\mu}_{\theta},\mathbf{\Sigma}_{\theta}) \Rightarrow P(\boldsymbol{\theta}_{i}) \propto \exp\left( \frac{1}{2}(\boldsymbol{\theta}_{i}\,-\,\boldsymbol{\mu}_{\theta})^{\top}\mathbf{\Sigma}_{\theta}^{-1}(\boldsymbol{\theta}_{i} \,-\,\boldsymbol{\mu}_{\theta})\right) \\ \boldsymbol{\mu}_{\theta} &\sim& N(\boldsymbol{\mu}_{0},\mathbf{V}_{0}) \!\Rightarrow\! P(\boldsymbol{\mu}_{\theta}) \propto \exp\left( \frac{1}{2}(\boldsymbol{\mu}_{\theta}\,-\,\boldsymbol{\mu}_{0})^{\top}\mathbf{V}_{0}^{-1}(\boldsymbol{\mu}_{\theta} \,-\, \boldsymbol{\mu}_{0})\right) \\ \mathbf{\Sigma}_{\theta}^{-1} &\sim& W(df_{0},\mathbf{S}_{0}) \end{array} $$

1.3 Likelihood

$$\begin{array}{@{}rcl@{}} L(\text{data},\{\boldsymbol{\theta}_{i}\},\boldsymbol{\mu}_{\theta},\mathbf{\Sigma}_{\theta})\,=\,P(\text{data}|\{\boldsymbol{\theta}_{i}\}) P(\{\boldsymbol{\theta}_{i}\}|\boldsymbol{\mu}_{\theta},\mathbf{\Sigma}_{\theta})P(\boldsymbol{\mu}_{\theta})P(\mathbf{\Sigma}_{\theta}), \end{array} $$

where P(data|{𝜃 _i }) corresponds to the Multinomial Logit model.

1.4 Full conditionals

$$\begin{array}{@{}rcl@{}} P(\boldsymbol{\theta}_{i}|\boldsymbol{\mu}_{\theta},\mathbf{\Sigma}_{\theta},\text{data}_{i}) \propto \exp\left( \,-\,\frac{1}{2}(\boldsymbol{\theta}_{i}\,-\,\boldsymbol{\mu}_{\theta})^{\top}\mathbf{\Sigma}_{\theta}^{-1}(\boldsymbol{\theta}_{i} \,-\, \boldsymbol{\mu}_{\theta})\right)P(\text{data}_{i}|\boldsymbol{\theta}_{i}) \end{array} $$

$$\boldsymbol{\mu}_{\theta} \sim N(\boldsymbol{\mu}_{i},\mathbf{V}_{i}), \mathbf{\Sigma}_{\theta}^{-1} \sim W(df_{1},\mathbf{S}_{1}) $$

where

$$\begin{array}{@{}rcl@{}} \mathbf{V}_{i}^{-1}&=&[\mathbf{V}_{0}^{-1}+N\mathbf{\Sigma}_{\theta}^{-1}]\\ \boldsymbol{\mu}_{i}&=&\mathbf{V}_{i}[\boldsymbol{\mu}_{0}\mathbf{V}_{0}^{-1}+N\bar{\theta}\mathbf{\Sigma}_{\theta}^{-1}]\\ df_{1} &=&df_{0}+N\\ \mathbf{S}_{1} &=& \sum\limits_{i=1}^{N}(\boldsymbol{\theta}_{i}-\boldsymbol{\mu}_{\theta})(\boldsymbol{\theta}_{i}-\boldsymbol{\mu}_{\theta})^{\top} +\mathbf{S}_{0}^{-1}. \end{array} $$

The MCMC procedure generates a sequence of draws from the posterior distribution of the model’s parameters. Since the full conditionals for 𝜃 _i do not have a closed form, the Metropolis-Hastings (M-H) algorithm is used to draw the samples. In particular, we use a Gaussian random-walk M-H where the proposal vector of parameters φ ^(t) for 𝜃 _i at iteration t is drawn from N(φ ^(t−1),σ ²Δ) and accepted using the M-H acceptance ratio. The tuning parameters σ and Δ are chosen adaptively to yield an acceptance rate of approximately 20 %.

We use the following uninformative prior hyperparameters: μ ₀=0, V ₀=10³ I _{N 𝜃×N 𝜃} , d f ₀ = N 𝜃+5, S ₀ = d f ₀ C, where N is the number of individuals, and C is an N 𝜃×N 𝜃 matrix with 2 on the diagonal and 1 off the diagonal for the levels of each attribute. We assume that the parameters are a priori uncorrelated across attributes (see e.g. [25]).

Appendix A: HB mixed logit estimation

In the proposed models, three parameters need to be calibrated: regularization parameter C, threshold 𝜖, and shrinkage 𝜃. We analyze how the performance of each model varies as a function of each parameter. For illustration purposes, we show the procedure used for the Camera data set. Similar analyses were conducted for the other data sets. Our goal was to assess whether the results are stable along different values of these parameters. A less rigorous validation strategy can be used in such a case. In contrast, a high variance in the performance requires an exhaustive model selection procedure such as LOOCV in order to find the best combination of parameters.

Figure 1 depicts the LOOCV hit rates as a function of C, 𝜖, and 𝜃 for the proposed feature selection approach.

Fig. 1

Leave-one-out validation hit rates for l ₁-SVM for different values of C, 𝜖, and 𝜃 (Camera data set)

Figure 1 reveals the influence of parameters C, 𝜖, and 𝜃 in the predictive performance (Leave-one-out validation hit rate). Results are relatively stable for small values of 𝜃 and 𝜖, and values of C around the unit, although we observe an important influence of these parameters in the final outcome of the proposed method.

Performing an adequate grid search is highly recommended, varying the parameters C, 𝜖, and 𝜃 along the suggested values in order to obtain the desired results. Additionally, the fact that the optimal values for these parameters are always above zero confirms the importance of feature selection and shrinkage to control for potential overfitting when a relatively small number of respondents is present.