Skip to main content
SpringerLink
Log in

Performance and Reproducibility of BERT4Rec

  • Conference paper
  • First Online:
New Trends in Database and Information Systems (ADBIS 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1850))

Included in the following conference series:

Abstract

Sequential Recommendation with Bidirectional Encoder Representations from Transformer, BERT4Rec, is an efficient and effective model for sequential recommendation, regarded as a state-of-the-art in this field. However, studies of said architecture achieve varying results, causing doubts whether it is possible to consistently reproduce the result with reported training parameters. This study aims to test the performance and reproducibility of a BERT4Rec implementation on MovieLens1M and Netflix Prize datasets. Overall findings suggest that while using a proper implementation, BERT4Rec can still be called a state-of-the-art solution, additional work is needed to increase reproducibility of the original model. Moreover, possible avenues for further improvement are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bennett, J., Lanning, S., et al.: The netflix prize. In: KDD cup and workshop. vol. 2007, p. 35. New York (2007)

    Google Scholar 

  2. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. preprint arXiv:1810.04805 (2018)

  3. Elahi, M., Ricci, F., Rubens, N.: A survey of active learning in collaborative filtering recommender systems. Comput. Sci. Rev. 20, 29–50 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  4. Fan, Z., Liu, Z., Zhang, J., Xiong, Y., Zheng, L., Yu, P.S.: Continuous-time sequential recommendation with temporal graph collaborative transformer. In: 30th ACM International Conference on Information and Knowledge Management, pp. 433–442 (2021)

    Google Scholar 

  5. Harper, F., Konstan, J.: Movielens data set. ACM Trans. Interact. Intell. Syst. 5(4), 1–19 (2006)

    Google Scholar 

  6. Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22, 5–53 (2004)

    Article  Google Scholar 

  7. Hidasi, B., Karatzoglou, A., Baltrunas, L., Tikk, D.: Session-based recommendations with recurrent neural networks. preprint arXiv:1511.06939 (2015)

  8. Jaywonchung: Pytorch implementation of bert4rec and netflix vae. https://github.com/jaywonchung/BERT4Rec-VAE-Pytorch

  9. Kang, W.C., McAuley, J.: Self-attentive sequential recommendation. In: 2018 IEEE ICDM, pp. 197–206. IEEE (2018)

    Google Scholar 

  10. Kannout, E.: Context clustering-based recommender systems. In: 2020 15th FedCSIS, pp. 85–91. IEEE (2020)

    Google Scholar 

  11. Karpus, A., Raczynska, M., Przybylek, A.: Things you might not know about the k-nearest neighbors algorithm. In: KDIR, pp. 539–547 (2019)

    Google Scholar 

  12. Koren, Y.: Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: 14th ACM SIGKDD, pp. 426–434 (2008)

    Google Scholar 

  13. Liang, D., Krishnan, R.G., Hoffman, M.D., Jebara, T.: Variational autoencoders for collaborative filtering (2018)

    Google Scholar 

  14. Lin, Z., et al.: A structured self-attentive sentence embedding. preprint arXiv:1703.03130 (2017)

  15. Lops, P., De Gemmis, M., Semeraro, G.: Content-based recommender systems: State of the art and trends. Recommender sys. handbook, pp. 73–105 (2011)

    Google Scholar 

  16. Ma, C., Kang, P., Liu, X.: Hierarchical gating networks for sequential recommendation. In: 25th ACM SIGKDD, pp. 825–833 (2019)

    Google Scholar 

  17. Paulina Leszczełowska, M.B., Grabski, M.: Systematic literature review on click through rate prediction. In: 2nd Workshop on PeRS at ADBIS’23 (2023)

    Google Scholar 

  18. Pazzani, M.J., Billsus, D.: Content-based recommendation systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web. LNCS, vol. 4321, pp. 325–341. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-72079-9_10

    Chapter  Google Scholar 

  19. Petrov, A., Macdonald, C.: Effective and efficient training for sequential recommendation using recency sampling. In: 16th ACM Conference on Recommender System (2022)

    Google Scholar 

  20. Petrov, A., Macdonald, C.: A systematic review and replicability study of bert4rec for sequential recommendation. In: 16th ACM Conference on Recommender System (2022)

    Google Scholar 

  21. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training (2018)

    Google Scholar 

  22. Rendle, S., Freudenthaler, C., Schmidt-Thieme, L.: Factorizing personalized markov chains for next-basket recommendation. In: 19th International Conference on World wide web, pp. 811–820 (2010)

    Google Scholar 

  23. Sarwar, B.M., Karypis, G., Konstan, J.A., Riedl, J.: Item-based collaborative filtering recommendation algorithms. In: The Web Conf. (2001)

    Google Scholar 

  24. Shenbin, I., Alekseev, A., Tutubalina, E., Malykh, V., Nikolenko, S.I.: Recvae: A new variational autoencoder for top-n recommendations with implicit feedback. In: 13th International Conference on Web Search and Data Mining. ACM (jan 2020). https://doi.org/10.1145/3336191.3371831

  25. Shiguang, W.: Furyton/recommender-baseline-model: Common used recommend baseline model. https://github.com/Furyton/Recommender-Baseline-Model

  26. Sun, F., et al.: Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In: 28th ACM International Conference on Information and Knowledge Management (2019)

    Google Scholar 

  27. Tang, J., Wang, K.: Personalized top-n sequential recommendation via convolutional sequence embedding. In: 11th ACM International Conference on Web Search and Data Mining, pp. 565–573 (2018)

    Google Scholar 

  28. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing System, vol. 30 (2017)

    Google Scholar 

  29. Yuan, F., Karatzoglou, A., Arapakis, I., Jose, J.M., He, X.: A simple convolutional generative network for next item recommendation. In: 12th ACM International Conference on Web Search and Data Mining, pp. 582–590 (2019)

    Google Scholar 

  30. Yue, Z., He, Z., Zeng, H., McAuley, J.: Black-box attacks on sequential recommenders via data-free model extraction. In: 15th ACM Conference on Recommender System (2021)

    Google Scholar 

  31. Zang, R., Zuo, M., Zhou, J., Xue, Y., Huang, K.: Sequence and distance aware transformer for recommendation systems. In: 2021 IEEE ICWS, pp. 117–124 (2021)

    Google Scholar 

  32. Zhao, W.X., et al.: Recbole 2.0: Towards a more up-to-date recommendation library. In: 31st ACM International Conference on Information & Knowledge Management, pp. 4722–4726 (2022)

    Google Scholar 

  33. Zhou, K., et al.: S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In: 29th ACM International Conference on Information & Knowledge Management, pp. 1893–1902 (2020)

    Google Scholar 

Download references

Acknowledgements

Special thanks to Shiguang Wu from Shandong University for sharing his implementation of BERT4Rec model.

Author information

Authors and Affiliations

  1. Gdansk University of Technology, 80-233, Gdansk, Poland

    Aleksandra Gałka, Jan Grubba & Krzysztof Walentukiewicz

Authors
  1. Aleksandra Gałka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jan Grubba
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Krzysztof Walentukiewicz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aleksandra Gałka .

Editor information

Editors and Affiliations

  1. Universitat Politècnica de Catalunya, Barcelona, Spain

    Alberto Abelló

  2. University of Ioannina, Ioannina, Greece

    Panos Vassiliadis

  3. Universitat Politècnica de Catalunya, Barcelona, Spain

    Oscar Romero

  4. Poznań University of Technology, Poznan, Poland

    Robert Wrembel

  5. University of Paris-Saclay, Gif-sur-Yvette, France

    Francesca Bugiotti

  6. Free University of Bozen-Bolzano, Bozen-Bolzano, Italy

    Johann Gamper

  7. CNRS, Villeurbanne Cedex, France

    Genoveva Vargas Solar

  8. University of Calabria, Rende, Italy

    Ester Zumpano

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gałka, A., Grubba, J., Walentukiewicz, K. (2023). Performance and Reproducibility of BERT4Rec. In: Abelló, A., et al. New Trends in Database and Information Systems. ADBIS 2023. Communications in Computer and Information Science, vol 1850. Springer, Cham. https://doi.org/10.1007/978-3-031-42941-5_55

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42941-5_55

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42940-8

  • Online ISBN: 978-3-031-42941-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation