Using neural word embeddings to model user behavior and detect user segments

Abstract

Modeling user behavior to detect segments of users to target and to whom address ads (behavioral targeting) is a problem widely-studied in the literature. Various sources of data are mined and modeled in order to detect these segments, such as the queries issued by the users. In this paper we first show the need for a user segmentation system to employ reliable user preferences, since nearly half of the times users reformulate their queries in order to satisfy their information need. Then we propose a method that analyzes the description of the items positively evaluated by the users and extracts a vector representation of the words in these descriptions (word embeddings). Since it is widely-known that users tend to choose items of the same categories, our approach is designed to avoid the so-called preference stability, which would associate the users to trivial segments. Moreover, we make sure that the interpretability of the generated segments is a characteristic offered to the advertisers who will use them. We performed different sets of experiments on a large real-world dataset, which validated our approach and showed its capability to produce effective segments.

Introduction

Behavioral targeting is the process of detecting segments of users with similar behaviors, in order to address effective ads to them. Given the high interest that extracting effective segments has, both the industry and the academia are studying ways to model user behavior. In the industry, the systems try to monitor the behavior of the users in implicit ways, in order to extract their preferences and form the segments; usually, neither the algorithms not the data are publicly made available, to avoid disclosing both industrial secrets and private information about the users. In the academic literature it has been highlighted that classic approaches to segmentation (like k-means) cannot take into account the semantics of the user behavior [1]. Tu and Lu [2] proposed a user segmentation approach based on a semantic analysis of the queries issued by the users, while Gong et al. [1] proposed a LDA-based semantic segmentation that groups users with similar query and click behaviors.

However, several problems remain open in the literature when performing a user segmentation by considering the user behavior.

Data sources reliability. In order to satisfy the users’ information need, query reformulation characterizes nearly 50% of the queries issued by the users [3], [4], [5]. Therefore, the semantic analysis of a query is not a reliable source of information, since it does not contain any information about whether or not a query led to what the user was really looking for. Moreover, performing a semantic analysis on the items evaluated by the users, in order to perform a filtering on them, can increase the accuracy of a system [6], [7], [8]. Considering these aspects, a possible solution to this issue would be a semantic analysis on the description of the items a user positively evaluated through an explicitly given rating. However, another issue arises in cascade.

Preference stability. The analysis of reliable information about the users, such as the description of the items evaluated by them, would probably lead to trivial segments, since users tend to evaluate items of the same categories (e.g., they usually watch movies of the same genres or by the same director/actor). This problem is known as preference stability [9] and on the one hand it leads to high-quality knowledge sources, while on the other hand there is no way to target the users with serendipitous and effective ads (overspecialization [10]).

Segmentation interpretability. Another open issue widely-studied in this research area is the capability for a segmentation to be easily interpreted. A recent survey on user segmentation (mostly focused on the library domain) [11], highlighted that, in order to create a proper segmentation of the users, it is important to understand them. On the one hand, easily interpretable approaches generate trivial segments, and even a partitioning with the k-means clustering algorithm has proven to be more effective than this method [12], while on the other hand, when a larger set of features is combined, the problem of properly understanding and interpreting results arises [13], [14]. This is mostly due to the lack of guidance on how to interpret the results of a segmentation [15]. The fact that easily understandable approaches generate ineffective segments, and that more complex ones are accurate but not easy to use in practice, generates an important gap in this research area.

Our contributions. In this paper, we present an approach to user segmentation, such that the sources of information used to build it are reliable, the generated user segmentation is not trivial and it is easily interpretable.

As previously mentioned, the problem of using reliable sources of information will be solved by considering the items positively evaluated by the users with an explicitly-assigned rating. In particular, we employ the vector representation of the words in a description (word embedding) [16]. Word embeddings are built by considering as input a text corpus, that leads to the building of a vocabulary, and to the learning of the vector representation of the words. They are largely employed nowadays in several NLP tasks, such as the representations of sentences and paragraphs [17], [18], relational entities [[19], [20], general text-based attributes [21], descriptive text of images [22], and nodes in graph structure [23]. According to the authors’ knowledge, no approach uses word embeddings for user segmentation purposes.

For each class of items that the users can be targeted with (e.g., movie genres), our approach builds a vector representation based on the word embeddings, which characterizes the words that represent the class. In a similar way, we also build a user model that captures the user interests. By matching the vector representations of a class of items with the user model, thanks to a similarity metric, we can associate a user to the segment that represents that class of items. It is trivial to notice that each user can have a strong similarity also with classes of items she never evaluated (thus avoiding the preference stability problem) and that the segments can be easily interpreted (even if a class and a user are represented by tens of features in the vector, the advertiser is only required to specify which interests she wants to target). In order to allow advertisers to specify more complex targets, we also present a Boolean algebra that combines multiple classes of items with simple operations (e.g., to extract a vector representation of what characterizes comedy AND romantic movies).

More formally, the problem statement is the following:

Problem 1

We are given a set of users $U=\{u_1,\dots ,u_N\},$ a set of items $I=\{i_1,\dots ,i_M\},$ and a set R of ratings used to express the user preferences (e.g., $R=[1,5]$ or $R=\{like,dislike\}$). The set of all possible preferences expressed by the users is a ternary relation P⊆U × I × R. We denote as $P_{+}\subseteq P$ the subset of preferences with a positive value, as $I_{+}$ the items for which there is a positive preference, and as I_u the items positively evaluated by a user u. The set of item descriptions is denoted as $D=\{d_1,\dots ,d_M\}$ (note that we have a description for each item, so $\mid D\mid =\mid I\mid$), and the vocabulary of the words in D is denoted as $V=\{v_1,\cdots ,v_W\}$. Let $NW{E}_{v_w}=\{l_1,\cdots ,l_Z\}$ be the vector representation (neural word embedding) of each word v_w ∈ V. We denote as $C=\{c_1,\dots ,c_K\}$ the set of primitive classes used to classify the items. Our first aim is to extract a vector representation of each class c_k ∈ C based on the neural word embeddings of the description of the items classified with c_k (neural class embedding, NCE), and a vector representation of each user u ∈ U (user model, m_u). The objective of this paper is to build a function f: C → U that, given a class, returns a set of users to target T⊆U, such that the similarity between the neural class embedding and the models of the users in T is higher than a threshold value.

The scientific contributions of our proposal are now presented:

• we propose a novel use of neural word embeddings for user segmentation purposes;

• we introduce a novel data structure, called NCE (Neural Class Embedding), able to model the words that characterize a class of items;

• we consider, for the first time in the user segmentation literature, the reliability of the data sources. Indeed, with respect to the literature that usually performs an analysis of the queries issued by the users, we rely on the description of the items a user positively rated;

• we avoid preference stability by considering the similarity between each user model and the vector representation of a classes of items, in order to allow the approach to include a user in a segment that represents a class of items she has never evaluated, but that is highly similar to her preferences;

• we present a Boolean algebra that allows us to specify, in a simple but punctual way, the interests that the segment should cover; the algebra, along with the built models, avoids the interpretability issues that usually characterize the segmentations based on several features.

The rest of the paper is organized as follows: we first present the works in the literature related with our approach (Section 2), then we continue with the implementation details (Section 3) and the description of the performed experiments (Section 4), ending with some concluding remarks (Section 5).