Abstract
Transformer and its variants have been intensively applied for sequential recommender systems nowadays as they take advantage of the self-attention mechanism, feed-forward network (FFN) and parallel computing capability to generate the high-quality sequence representation. Recently, a wide range of fast, efficient Transformers have been proposed to facilitate sequence modeling, however, the lack of a well-established benchmark might lead to the non-reproducible and even inconsistent results across different works, making it hard to gain rigorous assessments. In this paper, We provide a benchmark for reproducibility and present a comprehensive empirical study on various Transformer-based recommendation approaches, and key techniques or components in Transformers. Based on this study, we propose a hybrid effective and Efficient Transformer variant for sequential Recommendation (ETRec), which incorporates the scalable long- and short-term preference learning, blocks of items aggregating as interests, and parameter-efficient cross-layer sharing FFN. Extensive experiments on six public benchmark datasets demonstrate the advanced efficacy of the proposed approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Beltagy, I., Peters, M.E., Cohan, A.: Longformer: the long-document transformer. CoRR abs/2004.05150 (2020)
Choromanski, K.M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., et al.: Rethinking attention with performers. In: ICLR (2021)
Dacrema, M.F., Cremonesi, P., Jannach, D.: Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In: RecSys (2019)
Dai, Z., Lai, G., Yang, Y., Le, Q.: Funnel-transformer: filtering out sequential redundancy for efficient language processing. In: NeurIPS (2020)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL (2019)
Fan, X., Liu, Z., Lian, J., Zhao, W.X., et al.: Lighter and better: low-rank decomposed self-attention networks for next-item recommendation. In: SIGIR (2021)
Fan, Z., e al.: Sequential recommendation via stochastic self-attention. In: TheWebConf (2022)
Geng, S., Liu, S., Fu, Z., et al.: Recommendation as language processing (RLP): a unified pretrain, personalized prompt & predict paradigm (P5). In: RecSys (2022)
Harper, F.M., Konstan, J.A.: The movielens datasets: history and context. ACM Trans. Interact. Intell. Syst. (2016)
Hidasi, B., Karatzoglou, A., Baltrunas, L., Tikk, D.: Session-based recommendations with recurrent neural networks. In: ICLR (2016)
Kang, W.C., et al.: Self-attentive sequential recommendation. In: ICDM (2018)
Katharopoulos, A., Vyas, A., Pappas, N., Fleuret, F.: Transformers are RNNs: fast autoregressive transformers with linear attention. In: ICML (2020)
Krichene, W., Rendle, S.: On sampled metrics for item recommendation. In: SIGKDD (2020)
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: Albert: a lite bert for self-supervised learning of language representations. In: ICLR (2020)
Li, J., Ren, P., Chen, Z., Ren, Z., Lian, T., Ma, J.: Neural attentive session-based recommendation. In: CIKM (2017)
Li, M., Zhao, X., Lyu, C., Zhao, M., Wu, R., Guo, R.: MLP4Rec: a pure MLP architecture for sequential recommendations. In: IJCAI (2022)
Li, Y., Chen, T., Zhang, P.F., Yin, H.: Lightweight self-attentive sequential recommendation. In: CIKM (2021)
McAuley, J., Targett, C., Shi, Q., Van Den Hengel, A.: Image-based recommendations on styles and substitutes. In: SIGIR (2015)
Pietruszka, M., Borchmann, L., Gralinski, F.: Sparsifying transformer models with trainable representation pooling. In: ACL (2022)
Rendle, S., Freudenthaler, C., Schmidt-Thieme, L.: Factorizing personalized markov chains for next-basket recommendation. In: WWW (2010)
de Souza Pereira Moreira, G., Rabhi, S., et al.: Transformers4Rec: bridging the gap between nlp and sequential/session-based recommendation. In: RecSys (2021)
Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., et al.: BERT4Rec: sequential recommendation with bidirectional encoder representations from transformer. In: CIKM (2019)
Tang, J., Wang, K.: Personalized top-n sequential recommendation via convolutional sequence embedding. In: WSDM (2018)
Tay, Y., Bahri, D., Metzler, D., Juan, D.C., Zhao, Z., Zheng, C.: Synthesizer: rethinking self-attention for transformer models. In: ICML (2021)
Tay, Y., et al.: Long range arena: a benchmark for efficient transformers. In: ICLR (2020)
Tolstikhin, I., et al.: MLP-Mixer: an all-MLP architecture for vision. In: NeurIPS (2021)
Vaswani, A., et al.: Attention is all you need. In: NIPS (2017)
Wang, S., Li, B.Z., Khabsa, M., Fang, H., Ma, H.: Linformer: self-attention with linear complexity. arXiv preprint arXiv:2006.04768 (2020)
Wu, Y., Lian, D., Gong, N.Z., Yin, L., Yin, M., et al.: Linear-time self attention with codeword histogram for efficient recommendation. In: TheWebConf (2021)
Yang, Z., Dai, Z., Yang, Y., Carbonell, J.G., et al.: XLNet: generalized autoregressive pretraining for language understanding. In: NeurIPS (2019)
Yuan, F., Karatzoglou, A., Arapakis, I., Jose, J.M., He, X.: A simple convolutional generative network for next item recommendation. In: WSDM (2019)
Zaheer, M., et al.: Big bird: transformers for longer sequences. In: NeurIPS (2020)
Zhao, W.X., et al.: RecBole 2.0: towards a more up-to-date recommendation library. In: CIKM (2022)
Zhao, W.X., Lin, Z., Feng, Z., Wang, P., Wen, J.R.: A revisiting study of appropriate offline evaluation for top-n recommendation algorithms. ACM Trans. Inf. Syst. 41(2) (2022)
Zhao, W.X., Mu, S., Hou, Y., Lin, Z., et al.: RecBole: towards a unified, comprehensive and efficient framework for recommendation algorithms. In: CIKM (2021)
Zhou, K., Yu, H., Zhao, W.X., Wen, J.: Filter-enhanced MLP is all you need for sequential recommendation. In: TheWebConf (2022)
Acknowledgements
This work was partially supported by National Natural Science Foundation of China under Grant No. 62222215, Beijing Natural Science Foundation under Grant No. 4222027, and Beijing Outstanding Young Scientist Program under Grant No. BJJWZYJH012019100020098. Xin Zhao is the corresponding author.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Sun, W., Liu, Z., Fan, X., Wen, JR., Zhao, W.X. (2023). Towards Efficient and Effective Transformers for Sequential Recommendation. In: Wang, X., et al. Database Systems for Advanced Applications. DASFAA 2023. Lecture Notes in Computer Science, vol 13944. Springer, Cham. https://doi.org/10.1007/978-3-031-30672-3_23
Download citation
DOI: https://doi.org/10.1007/978-3-031-30672-3_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30671-6
Online ISBN: 978-3-031-30672-3
eBook Packages: Computer ScienceComputer Science (R0)