Enhancing Sequence Representation for Personalized Search

Abstract

The critical process of personalized search is to reorder candidate documents of the current query based on the user’s historical behavior sequence. There are many types of information contained in user historical information sequence, such as queries, documents, and clicks. Most existing personalized search approaches concatenate these types of information to get an overall user representation, but they ignore the associations among them. We believe the associations of different information mentioned above are significant to personalized search. Based on a hierarchical transformer as base architecture, we design three auxiliary tasks to capture the associations of different information in user behavior sequence. Under the guidance of mutual information, we adjust the training loss, enabling our PSMIM model to better enhance the information representation in personalized search. Experimental results demonstrate that our proposed method outperforms some personalized search methods.

Appendices

Appendix 1 Mutual Information Maximization

Mutual information maximization is a pivotal strategy for integrating diverse forms of historical information. Rooted in information theory, mutual information (MI) is a valuable tool for quantifying the dependence between random variables. Its mathematical definition is expressed as:

$$\begin{aligned} I(A,B) = P(A) - P(A|B) = P(B) - P(B|A). \end{aligned}$$

(17)

Suppose A and B are different views of the input data, such as a word and its context in NLP tasks or a document and its historical context sequence in personalized search. Let f be a function receiving $A=a$ and $B=b$ as inputs. The primary aim of maximizing MI is to tune the parameters of f to maximize the mutual information I(A, B), thereby extracting the most discriminative and salient attributes of the samples.

The essence of effective feature extraction involves distinguishing a sample from the entire dataset by capturing its distinctive information. By maximizing mutual information, one can isolate and harness such unique characteristics. However, when f constitutes neural networks or other encoders, directly optimizing MI is usually tricky [19]. Thus, a common workaround is to find a tractable lower bound for I(A, B) that closely approximates the target function. A specific lower bound proved to be effective in practice is InfoNCE [14, 18], which is based on noise contrast estimation [9]. InfoNCE is defined as follows:

$$\begin{aligned} \textrm{InfoNCE} = \mathbb {E}_p(A,B) \left( f_\theta (a,b)-\mathbb {E}_{q( \tilde{\mathcal {B}} )} \left( log\sum _{\tilde{b} \in {\tilde{\mathcal {B}} }} {\textrm{exp} f_\theta (a,\tilde{b})} \right) \right) + log|\tilde{\mathcal {B}}|, \end{aligned}$$

(18)

where a and b are different views of the input data, and $f_{\theta }\in {\mathbb {R}}$ is a function whose parameter is $\theta$ (for example, dot product result expressed by word and context or cos distance). $\tilde{\mathcal {B}}$ is a set of samples taken from the distribution $q(\tilde{\mathcal {B}})$. The B set contains a positive sample b and $|\tilde{\mathcal {B}}| - 1$ negative samples. Learning representation based on this goal is also called contrastive learning.

We can see that the InfoNCE is analogous to the cross-entropy form the formula below when $\tilde{\mathcal {B}}$ can take all possible values of B (i.e., $\tilde{\mathcal {B}}=\mathcal {B}$) and they are uniformly distributed, maximizing InfoNCE is analogous to maximize the cross-entropy loss:

$$\begin{aligned} \mathbb {E}_p(A,B) = \left( f_\theta (a,b)- \textrm{log}\sum _{\tilde{b} \in {\mathcal {B} }} {\textrm{exp} f_\theta (a,\tilde{b})} \right) . \end{aligned}$$

(19)

Appendix 2 Implementation Details

The parameters of our model PSMIM are set as follows: The word embedding size is 100. The hidden size of the transformer layer in our base model is 512. The number of heads in multi-head attention is 8. The size of the MLP hidden layer is 256. In the experiment, we set the hyperparameters $z_1$, $z_2$ and $z_3$ in Formula 15 of three auxiliary tasks to 1.0. We use the Adam optimizer to minimize the final loss $L_\textrm{total}$, and the learning rate of our optimizer is $1e^{-3}$. In the experiment, we set $\alpha$ in Formula 16 to 1.0. In addition, the number of matched cores for the KRNM model is 11.