Mend the Learning Approach, Not the Data: Insights for Ranking E-Commerce Products

Abstract

Improved search quality enhances users’ satisfaction, which directly impacts sales growth of an E-Commerce (E-Com) platform. Traditional Learning to Rank (LTR) algorithms require relevance judgments on products. In E-Com, getting such judgments poses an immense challenge. In the literature, it is proposed to employ user feedback (such as clicks, add-to-basket (AtB) clicks and orders) to generate relevance judgments. It is done in two steps: first, query-product pair data are aggregated from the logs and then order rate etc. are calculated for each pair in the logs. In this paper, we advocate counterfactual risk minimization (CRM) approach which circumvents the need of relevance judgements, data aggregation and is better suited for learning from logged data, i.e. contextual bandit feedback. Due to unavailability of public E-Com LTR dataset, we provide Mercateo dataset from our platform. It contains more than 10 million AtB click logs and 1 million order logs from a catalogue of about 3.5 million products associated with 3060 queries. To the best of our knowledge, this is the first work which examines effectiveness of CRM approach in learning ranking model from real-world logged data. Our empirical evaluation shows that our CRM approach learns effectively from logged data and beats a strong baseline ranker ($\lambda$-MART) by a huge margin. Our method outperforms full-information loss (e.g. cross-entropy) on various deep neural network models. These findings demonstrate that by adopting CRM approach, E-Com platforms can get better product search quality compared to full-information approach.

Appendices

A Comparison of Counterfactual Risk Estimators

We compare the performance of SNIPS estimator with two baseline estimators for counterfactual risk. We conduct the experiments on AtB click training data of Mercateo dataset. The inverse porpensity scoring (IPS) estimator is calculated by:

$$\begin{aligned} \hat{R}_{IPS}(\pi _w) = \frac{1}{n}\sum _{i=1}^n \delta _i \frac{\pi _w(a_i|c_i)}{\pi _0(a_i|c_i)}. \end{aligned}$$

(4)

Second estimator is an empirical average (EA) estimator defined as follows:

$$\begin{aligned} \hat{R}_{EA}(\pi _w) = \sum _{(c,a) \in (\mathcal {C,A})} \overline{\delta } (c,a) \pi _w (a|c) , \end{aligned}$$

(5)

where $\overline{\delta } (c,a)$ is the empirical average of the losses for a given context and action pair. The results for these estimators are provided in Table 5. Compared to SNIPS both IPS and EA perform significantly worse on all evaluated metrics. The results confirm the importance of equivariance of the counterfactual estimator and show the advantages of SNIPS estimator.

Table 5. Results on Mercateo dataset with AtB click relevance for IPS and empirical average estimators

Fig. 3.

SNIPS denominator vs $\lambda$ on order logs (training set)

Fig. 4.

Performance on orders test set of rankers trained with different $\lambda$

B Choosing Hyperparameter $\lambda$

One major drawback of SNIPS estimator is that, being a ratio estimator, it is not possible to perform its direct stochastic optimization [16]. In particular, given the success of stochastic gradient descent (SGD) training of deep neural networks in related applications, this is quite disadvantageous as one can not employ SGD for training.

To overcome this limitation, Joachims et al. [16] fixed the value of denominator in Eq. 3. They denote the denominator by S and solve multiple constrained optimization problems for different values of S. Each of these problems can be reformulated using lagrangian of the constrained optimization problem as:

$$\begin{aligned} \hat{w}_j = \mathop {\text {argmin}}\limits _{w} \frac{1}{n} \sum _{i=1}^n (\delta _i - \lambda _j) \frac{\pi _w(a_i|c_i)}{\pi _0(a_i|c_i)} \end{aligned}$$

(6)

where $\lambda _j$ corresponds to a fixed denominator $S_j$.

The main difficulty in applying the CRM method to learn from logged data is the need to choose hyperparameter $\lambda$. We discuss below our heuristics of selecting it. We also evaluate the dependence of $\lambda$ on SNIPS denominator S, which can be used to guide the search for $\lambda$. To achieve good performance with CRM loss, one has to tune hyperparameter $\lambda \in [0, 1]$. Instead of doing a grid search, we follow a smarter way to find a suitable $\lambda$. Building on the observations proposed in [16], we can guide the search of $\lambda$ based on value of SNIPS denominator S. It was shown in [16] that the value of S increases monotonically, if $\lambda$ is increased. Secondly, it is straightforward to note that expectation of S is 1. This implies that, with increasing number of bandit feedback, the optimal value for $\lambda$ should be selected such that its corresponding S value concentrates around 1. In our experiments, we first select some random $\lambda \in [0, 1]$ and train the model for two epochs with this $\lambda$. We then calculate S for the trained model; if S is greater than 1, we decrease $\lambda$ by 10%, otherwise we increase it by 10%. The final value of $\lambda$ is decided based on best performance on validation set.

In Fig. 3, we plot the values of denominator S on order logs (training set) of Mercateo dataset for different values of hyperparameter $\lambda$. On the figure below, Fig. 4, we also plot performance on orders test set, in terms of MAP and NDCG@5 scores, of different rankers for these values of hyperparameter $\lambda$. It is to be noted that the values of SNIPS denominator S monotonically increase with increasing $\lambda$. The MAP and NDCG@5 reach its highest value for $\lambda = 0.4$, but decrease only slightly with increasing values of $\lambda$. Furthermore, it can also be seen from these two figures that the $\lambda$ values with good performance on test set have corresponding SNIPS denominator values close to 1.