A systematic framework of predicting customer revisit with in-store sensors

Abstract

Recently, there is a growing number of off-line stores that are willing to conduct customer behavior analysis. In particular, predicting revisit intention is of prime importance, because converting first-time visitors to loyal customers is very profitable. Thanks to noninvasive monitoring, shopping behaviors and revisit statistics become available from a large proportion of customers who turn on their mobile devices. In this paper, we propose a systematic framework to predict the revisit intention of customers using Wi-Fi signals captured by in-store sensors. Using data collected from seven flagship stores in downtown Seoul, we achieved 67–80% prediction accuracy for all customers and 64–72% prediction accuracy for first-time visitors. The performance improvement by considering customer mobility was 4.7–24.3%. Furthermore, we provide an in-depth analysis regarding the effect of data collection period as well as visit frequency on the prediction performance and present the robustness of our model on missing customers. We released some tutorials and benchmark datasets for revisit prediction at https://github.com/kaist-dmlab/revisit.

Appendices

A. Comparison on various classifiers

We compared the performances of eight classifiers. We used default parameter settings for classifiers and some tuned parameters are listed below.

• Classifiers provided by Scikit-learn  [26].¹¹ The parameters used are summarized as follows.

  • LR (Logistic Regression): default settings.

  • DT (Decision Tree): max_depth = 4.

  • RF (Random Forests): n_estimator = 10.

  • AB (AdaBoost): default settings.

  • GB (Gradient Boosting): max_depth = 4.

• Up-to-date boosting classifiers:

  • CAB (CatBoost): depth = 4, learning_rate = 0.1, iterations = 30.

  • XGB (XGBoost): max_depth = 4, learning_rate = 0.1.

  • LGB (LightGBM): max_depth = 4, learning_rate = 0.1.

Fig. 13

Comparison between classifiers. LGB turns out to be the most effective among all classifiers. a Average accuracy on all experiments, b average running time on all experiments

Figure 13 summarizes the comparison results for the eight classifiers in terms of prediction accuracy and running time. To obtain stable results, we repeated fivefold cross-validation 25 times and then reported the averages by aggregating the results of the seven stores. As a result, LGB turned out to be the fastest classifier among the three best-performing classifiers—GB, XGB, and LGB. CAB was very fast as well as gave comparable results. Interestingly, DT took more time than RF and showed a better result in the default setting. Table 8 shows the details of Fig. 13 by showing the accuracy for each of the seven stores. The mean and standard deviation were calculated from the average accuracies of 25 different fivefold cross-validations.

Table 8 Prediction accuracy (%) of various classifiers for the revisit prediction task

B. Comparison on stacking models

To achieve additional performance improvement, we applied stacking (meta ensembling) with eight strategies. Stacking is a model ensembling technique used to combine multiple predictive models to generate a better model  [38]. Usually, the stacked model is known to outperform each of the individual models owing to its smoothing nature and its ability to highlight each base model. The main point of the stacking is to utilize the prediction results of the base models as features for the stacking model in the second layer.

To do this, we selected CAB, XGB, and LGB as the base models. We further separated a training set into three subsets and used two subsets to make the prediction labels for the remaining subset. The prediction labels for the testing set were also calculated together three ($=_3\!C_2$) times, and the three sets of the labels for the testing set were averaged for the final use. In this way, we generated the label features for both training and testing sets. These additional features are fed to the final LGB stacking model. We followed a general procedure from the reference¹² and added three options. Figure 14 illustrates the process of creating eight stacking models ($M_1$–$M_8$) through the choice of the three options. The description of the three options is as follows.

• Sampling strategy: A parameter that determines whether to use either random oversampling  [21] or downsampling. This option is not directly related to the stacking, but we added it to improve the accuracy by treating the class imbalance problem.

• # of predictions: A parameter that determines whether to use one model or multiple models for each fold. The former case generates a single additional feature, and the latter case generates three additional features.

• Using only labels: A parameter that determines whether to use only the prediction labels (one or three features) or to use all existing features with the prediction labels (n+1 or n+3 features where n is the total number of hand-engineered features used).

Fig. 14

Stacking options

Table 9 Prediction accuracy (%) of stacking models for the revisit prediction task with the data of all visitors

Table 9 shows the average accuracy results obtained for each of the seven stores in details.¹³ We observed that the performance improvement was not so high despite the long running time of the stacking model. Thus, we conjecture that each of the best-performing classifiers achieved almost the highest accuracy by itself.

C. Lower bounds of prediction accuracy

The visit logs $v_k$ with the same visit count k are considered to have the same information. To maximize the accuracy, we must predict the label l of $v_k$ by the following criteria:

$$\begin{aligned} \forall v: l(v \in v_k)= {\left\{ \begin{array}{ll} 1, &{} \text {if } E[RV_{\mathrm{bin}}(v_k)] \ge 1/2\\ 0, &{} \text {otherwise}. \end{array}\right. } \end{aligned}$$

(2)

Considering each proportion $p_k = |v_k|/\sum _{k}{|v_k|}$ and simplifying $E[RV_{\mathrm{bin}}(v_k)]$ as $r_k$, the lower bound accuracy of a model can be represented as $LB = \sum _{k}p_k \cdot \max (r_k, 1-r_k)$. In the experiment of only first-time visitors, $LB = 1/2$ since $p_1 = 1$ and $r_1 = 1/2$.

The interpretation with the lower bound is as follows. For higher predictability, the revisit tendency of each $v_k$ should be homogeneous. In Fig. 15, we can notice that store L_MD is more predictable than A_GN, because $|r_k-0.5|$ of L_MD is larger than that of A_GN for the majority of k.

Fig. 15

Lower bound accuracies of two stores

D. Assumptions to interpret the data

Here, we would like to clarify how we count the first-time visitors and explain several underlying assumptions to consider.

• Assumption 1: Because we do not know whether customers visited a store before data was collected, we simply assume that the customers did not visit before the collection period. We believe that this assumption is reasonable because the stores in which we collected the data were relatively new at that time we began data collection.

• Assumption 2: Because customers are captured only when they turn on the Wi-Fi of their mobile device, we assume that the customers’ Wi-Fi turn on behavior is consistent when they visit the store. Also, we assume that there is no correlation between Wi-Fi usage and customer groups (first-time visitors and VIP customers).

• Assumption 3: We assume that customers visit the store with a device having the same MAC address. For this purpose, we retained only Android devices but removed Apple devices in the preprocessing step, because the later versions of iOS 8.0 follow a MAC-address randomization policy  [21] which makes infeasible to identify the same customer.

Rigorously speaking, the proportion of true first-time visitors would be less than 70% by considering all the effects explained above. Nevertheless, these customers are also likely to be early stage visitors.

E. Deciding the group movement threshold

We decided 30 s group movement threshold by the following logic. According to our observation at store E_GN in the afternoon of June 24 and June 26, 2017, 56% of 105 customers entered the store with their companions, which was more than half. Considering $p_x=39.2\%$ as the on-site Wi-Fi turn on rate (Always-on: 29.2%, Conditionally-on: 10%)  [24] and $p_y = 56\%$ as the actual proportion of customers in a group, we expected that $p_{yo}=15.5\%$ of the total visitors were represented as having companions in our collected data of store E_GN (by Eq. 1 in Sect. 5.3.2). By setting 30 s as a threshold of accompaniment, we also obtained 15% of the total visitors were considered as having companions in the same data. By considering a gap between actual group ratio and observed group ratio, we claim that 30 s is an appropriate threshold to distinguish group movement.