Skip to main content

Sequential Nature of Recommender Systems Disrupts the Evaluation Process

  • Conference paper
  • First Online:
Advances in Bias and Fairness in Information Retrieval (BIAS 2022)

Abstract

Datasets are often generated in a sequential manner, where the previous samples and intermediate decisions or interventions affect subsequent samples. This is especially prominent in cases where there are significant human-AI interactions, such as in recommender systems. To characterize the importance of this relationship across samples, we propose to use adversarial attacks on popular evaluation processes. We present sequence-aware boosting attacks and provide a lower bound on the amount of extra information that can be exploited from a confidential test set solely based on the order of the observed data. We use real and synthetic data to test our methods and show that the evaluation process on the MovieLense-100k dataset can be affected by \(\sim \)1% which is important when considering the close competition. Codes are publicly available (https://github.com/alishiraliGit/augmented-boosting-attack).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://paperswithcode.com/sota/collaborative-filtering-on-movielens-100k.

References

  1. Blum, A., Hardt, M.: The ladder: a reliable leaderboard for machine learning competitions. In: International Conference on Machine Learning, pp. 1006–1014. PMLR (2015)

    Google Scholar 

  2. Garcin, F., Dimitrakakis, C., Faltings, B.: Personalized news recommendation with context trees. In: Proceedings of the 7th ACM Conference on Recommender Systems, pp. 105–112 (2013)

    Google Scholar 

  3. Harper, F.M., Konstan, J.A.: The movielens datasets: history and context. ACM Trans. Interact. Intell. Syst. (TIIS) 5(4), 1–19 (2015)

    Google Scholar 

  4. He, R., McAuley, J.: Fusing similarity models with Markov chains for sparse sequential recommendation. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 191–200. IEEE (2016)

    Google Scholar 

  5. Hernández-Lobato, J.M., Houlsby, N., Ghahramani, Z.: Probabilistic matrix factorization with non-random missing data. In: International Conference on Machine Learning, pp. 1512–1520. PMLR (2014)

    Google Scholar 

  6. Hidasi, B., Karatzoglou, A., Baltrunas, L., Tikk, D.: Session-based recommendations with recurrent neural networks. arXiv:1511.06939 (2015)

  7. Little, R.J., Rubin, D.B.: Statistical Analysis With Missing Data, vol. 793. John Wiley & Sons, Hoboken (2019)

    Google Scholar 

  8. Ma, W., Chen, G.H.: Missing not at random in matrix completion: The effectiveness of estimating missingness probabilities under a low nuclear norm assumption. arXiv:1910.12774 (2019)

  9. Pradel, B., Usunier, N., Gallinari, P.: Ranking with non-random missing ratings: influence of popularity and positivity on evaluation metrics. In: Proceedings of the Sixth ACM Conference on Recommender Systems, pp. 147–154 (2012)

    Google Scholar 

  10. Rendle, S., Freudenthaler, C., Schmidt-Thieme, L.: Factorizing personalized markov chains for next-basket recommendation. In: Proceedings of the 19th International Conference on World Wide Web, pp. 811–820 (2010)

    Google Scholar 

  11. Schnabel, T., Swaminathan, A., Singh, A., Chandak, N., Joachims, T.: Recommendations as treatments: debiasing learning and evaluation. In: International Conference on Machine Learning, pp. 1670–1679. PMLR (2016)

    Google Scholar 

  12. Shani, G., Heckerman, D., Brafman, R.I., Boutilier, C.: An MDP-based recommender system. J. Mach. Learn. Res. 6(9), 1 (2005)

    Google Scholar 

  13. Wang, X., Zhang, R., Sun, Y., Qi, J.: Doubly robust joint learning for recommendation on data missing not at random. In: International Conference on Machine Learning, pp. 6638–6647. PMLR (2019)

    Google Scholar 

  14. Wu, C.Y., Ahmed, A., Beutel, A., Smola, A.J., Jing, H.: Recurrent recommender networks. In: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pp. 495–503 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ali Shirali .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shirali, A. (2022). Sequential Nature of Recommender Systems Disrupts the Evaluation Process. In: Boratto, L., Faralli, S., Marras, M., Stilo, G. (eds) Advances in Bias and Fairness in Information Retrieval. BIAS 2022. Communications in Computer and Information Science, vol 1610. Springer, Cham. https://doi.org/10.1007/978-3-031-09316-6_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-09316-6_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-09315-9

  • Online ISBN: 978-3-031-09316-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics