Abstract
A crucial part of recommender systems is to model the user’s preference based on her previous interactions. Different neural networks (e.g., Recurrent Neural Networks), that predict the next item solely based on the sequence of interactions have been successfully applied to sequential recommendation. Recently, BERT4Rec has been proposed, which adapts the BERT architecture based on the Transformer model and training methods used in the Neural Language Modeling community to this task. However, BERT4Rec still only relies on item identifiers to model the user preference, ignoring other sources of information. Therefore, as a first step to include additional information, we propose KeBERT4Rec, a modification of BERT4Rec, which utilizes keyword descriptions of items. We compare two variants for adding keywords to the model on two datasets, a Movielens dataset and a dataset of an online fashion store. First results show that both versions of our model improves the sequential recommending task compared to BERT4Rec.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
- 2.
We only adapted the batch size to our hardware restrictions and increased the number of epochs for training, because first experiments indicated that our models need more training time. Our code is available at https://dmir.org/KeBERT4Rec.
- 3.
We train all models on the ML-20m for 200 epochs. Our numbers for BERT4Rec are better than the ones reported in [7], as they train shorter.
References
Chen, Q., Zhao, H., Li, W., Huang, P., Ou, W.: Behavior sequence transformer for e-commerce recommendation in Alibaba. In: Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data. ACM, August 2019. https://doi.org/10.1145/3326937.3341261
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota, June 2019. https://doi.org/10.18653/v1/N19-1423
Hendrycks, D., Gimpel, K.: Gaussian error linear units (GELUs) (2016). http://arxiv.org/abs/1606.08415, cite arxiv:1606.08415. Comment: Trimmed version of 2016 draft
Hidasi, B., Quadrana, M., Karatzoglou, A., Tikk, D.: Parallel recurrent neural network architectures for feature-rich session-based recommendations. In: Proceedings of the 10th ACM Conference on Recommender Systems. RecSys 2016, pp. 241–248. ACM, New York (2016). https://doi.org/10.1145/2959100.2959167
Hidasi, B., Karatzoglou, A., Baltrunas, L., Tikk, D.: Session-based recommendations with recurrent neural networks. In: Bengio, Y., LeCun, Y. (eds.) ICLR (Poster) (2016)
Kang, W.C., McAuley, J.: Self-attentive sequential recommendation. In: 2018 IEEE International Conference on Data Mining (ICDM), pp. 197–206. IEEE (2018)
Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., Jiang, P.: BERT4Rec: sequential recommendation with bidirectional encoder representations from transformer. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management - CIKM 2019. ACM Press (2019). https://doi.org/10.1145/3357384.3357895
Tang, J., Wang, K.: Personalized top-n sequential recommendation via convolutional sequence embedding. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining - WSDM 2018. ACM Press (2018). https://doi.org/10.1145/3159652.3159656
Taylor, W.L.: “cloze procedure”: a new tool for measuring readability. J. Mass Commun. Quart. 30, 415–433 (1953)
Tuan, T.X., Phuong, T.M.: 3D convolutional networks for session-based recommendation with content features. In: Proceedings of the Eleventh ACM Conference on Recommender Systems, RecSys 2017, pp. 138–146. ACM, New York (2017). https://doi.org/10.1145/3109859.3109900
Vaswani, A., et al.: Attention is all you need. In: Advances in neural information processing systems, pp. 5998–6008 (2017)
Xu, C., Zhao, P., Liu, Y., Xu, J., S.Sheng, V.S., Cui, Z., Zhou, X., Xiong, H.: Recurrent convolutional neural network for sequential recommendation. In: The World Wide Web Conference - WWW 2019. ACM Press (2019). https://doi.org/10.1145/3308558.3313408
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Fischer, E., Zoller, D., Dallmann, A., Hotho, A. (2020). Integrating Keywords into BERT4Rec for Sequential Recommendation. In: Schmid, U., Klügl, F., Wolter, D. (eds) KI 2020: Advances in Artificial Intelligence. KI 2020. Lecture Notes in Computer Science(), vol 12325. Springer, Cham. https://doi.org/10.1007/978-3-030-58285-2_23
Download citation
DOI: https://doi.org/10.1007/978-3-030-58285-2_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58284-5
Online ISBN: 978-3-030-58285-2
eBook Packages: Computer ScienceComputer Science (R0)