Abstract
Datasets are often generated in a sequential manner, where the previous samples and intermediate decisions or interventions affect subsequent samples. This is especially prominent in cases where there are significant human-AI interactions, such as in recommender systems. To characterize the importance of this relationship across samples, we propose to use adversarial attacks on popular evaluation processes. We present sequence-aware boosting attacks and provide a lower bound on the amount of extra information that can be exploited from a confidential test set solely based on the order of the observed data. We use real and synthetic data to test our methods and show that the evaluation process on the MovieLense-100k dataset can be affected by \(\sim \)1% which is important when considering the close competition. Codes are publicly available (https://github.com/alishiraliGit/augmented-boosting-attack).
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Blum, A., Hardt, M.: The ladder: a reliable leaderboard for machine learning competitions. In: International Conference on Machine Learning, pp. 1006–1014. PMLR (2015)
Garcin, F., Dimitrakakis, C., Faltings, B.: Personalized news recommendation with context trees. In: Proceedings of the 7th ACM Conference on Recommender Systems, pp. 105–112 (2013)
Harper, F.M., Konstan, J.A.: The movielens datasets: history and context. ACM Trans. Interact. Intell. Syst. (TIIS) 5(4), 1–19 (2015)
He, R., McAuley, J.: Fusing similarity models with Markov chains for sparse sequential recommendation. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 191–200. IEEE (2016)
Hernández-Lobato, J.M., Houlsby, N., Ghahramani, Z.: Probabilistic matrix factorization with non-random missing data. In: International Conference on Machine Learning, pp. 1512–1520. PMLR (2014)
Hidasi, B., Karatzoglou, A., Baltrunas, L., Tikk, D.: Session-based recommendations with recurrent neural networks. arXiv:1511.06939 (2015)
Little, R.J., Rubin, D.B.: Statistical Analysis With Missing Data, vol. 793. John Wiley & Sons, Hoboken (2019)
Ma, W., Chen, G.H.: Missing not at random in matrix completion: The effectiveness of estimating missingness probabilities under a low nuclear norm assumption. arXiv:1910.12774 (2019)
Pradel, B., Usunier, N., Gallinari, P.: Ranking with non-random missing ratings: influence of popularity and positivity on evaluation metrics. In: Proceedings of the Sixth ACM Conference on Recommender Systems, pp. 147–154 (2012)
Rendle, S., Freudenthaler, C., Schmidt-Thieme, L.: Factorizing personalized markov chains for next-basket recommendation. In: Proceedings of the 19th International Conference on World Wide Web, pp. 811–820 (2010)
Schnabel, T., Swaminathan, A., Singh, A., Chandak, N., Joachims, T.: Recommendations as treatments: debiasing learning and evaluation. In: International Conference on Machine Learning, pp. 1670–1679. PMLR (2016)
Shani, G., Heckerman, D., Brafman, R.I., Boutilier, C.: An MDP-based recommender system. J. Mach. Learn. Res. 6(9), 1 (2005)
Wang, X., Zhang, R., Sun, Y., Qi, J.: Doubly robust joint learning for recommendation on data missing not at random. In: International Conference on Machine Learning, pp. 6638–6647. PMLR (2019)
Wu, C.Y., Ahmed, A., Beutel, A., Smola, A.J., Jing, H.: Recurrent recommender networks. In: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pp. 495–503 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Shirali, A. (2022). Sequential Nature of Recommender Systems Disrupts the Evaluation Process. In: Boratto, L., Faralli, S., Marras, M., Stilo, G. (eds) Advances in Bias and Fairness in Information Retrieval. BIAS 2022. Communications in Computer and Information Science, vol 1610. Springer, Cham. https://doi.org/10.1007/978-3-031-09316-6_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-09316-6_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-09315-9
Online ISBN: 978-3-031-09316-6
eBook Packages: Computer ScienceComputer Science (R0)