Abstract
Due to the applications of user engagements in recommender systems, predicting user engagement has recently attracted considerable attention. In this task which is firstly proposed in ACM Recommender Systems Challenge 2014, the posts containing users’ opinions about items (e.g., the tweets containing the users’ ratings about movies in the IMDb website) are studied. In this paper, we focus on user engagement evaluation for cold-start web applications in the extreme case, when there is no training data available for the target web application. We propose an adaptive model based on transfer learning (TL) technique to train on the data from a web application and test on another one. We study the problem of detecting tweets with positive engagement, which is a highly imbalanced classification problem. Therefore, we modify the loss function of the employed transfer learning method to cope with imbalanced data. We evaluate our method using a dataset including the tweets of four popular and diverse data sources, i.e., IMDb, YouTube, Goodreads, and Pandora. The experimental results show that in some cases transfer learning can transfer knowledge among domains to improve the user engagement evaluation performance. We further analyze the results to figure out when transfer learning can help to improve the performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
In each tweets, the user gives a rate to or likes/dislikes a product.
- 2.
“User Engagement as Evaluation” Challenge, http://2014.recsyschallenge.com/.
- 3.
There are lots of tweets with zero engagement and a few tweets with positive engagement.
- 4.
The dataset is freely available at http://ece.ut.ac.ir/node/100770.
- 5.
The results without instance weighting is biased toward the majority class. For the sake of space, the results without instance weighting are not reported.
References
Akbani, R., Kwek, S.S., Japkowicz, N.: Applying support vector machines to imbalanced datasets. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 39–50. Springer, Heidelberg (2004)
Loiacono, D., Lommatzsch, A., Turrin, R.: An Analysis of the 2014 RecSys challenge. In: RecSysChallenge, pp. 1–6 (2014)
Long, M., Wang, J., Ding, G., Pan, S.J., Yu, P.S.: Adaptation regularization: a general framework for transfer learning. IEEE Trans. Knowl. Data Eng. 26(5), 1076–1089 (2014)
Petrovic, S., Osborne, M., Lavrenko, V.: RT to win! predicting message propagation in twitter. In: ICWSM, pp. 586–589 (2011)
Powers, D.: Evaluation: from precision, recall and F-measure to ROC, informedness, markedness & correlation. J. Mach. Learn. Tech. 2(1), 37–63 (2011)
Said, A., Dooms, S., Loni, B., Tikk, D.: Recommender systems challenge 2014. In: RecSys, pp. 387–388 (2014)
de Souza, J.G.C., Zamani, H., Negri, M., Turchi, M., Falavigna, D.: Multitask learning for adaptive quality estimation of automatically transcribed utterances. In: NAACL-HLT, pp. 714–724 (2015)
Uysal, I., Croft, W.B.: User oriented tweet ranking: a filtering approach to microblogs. In: CIKM, pp. 2261–2264 (2011)
Zamani, H., Moradi, P., Shakery, A.: Adaptive user engagement evaluation via multi-task learning. In: SIGIR, pp. 1011–1014 (2015)
Zamani, H., Shakery, A., Moradi, P.: Regression and learning to rank aggregation for user engagement evaluation. In: RecSysChallenge, pp. 29–34 (2014)
Acknowledgements
This work was supported in part by the Center for Intelligent Information Retrieval. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Montazeralghaem, A., Zamani, H., Shakery, A. (2016). Cross Domain User Engagement Evaluation. In: Ferro, N., et al. Advances in Information Retrieval. ECIR 2016. Lecture Notes in Computer Science(), vol 9626. Springer, Cham. https://doi.org/10.1007/978-3-319-30671-1_64
Download citation
DOI: https://doi.org/10.1007/978-3-319-30671-1_64
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-30670-4
Online ISBN: 978-3-319-30671-1
eBook Packages: Computer ScienceComputer Science (R0)