skip to main content
10.1145/3589335.3651527acmconferencesArticle/Chapter ViewAbstractPublication PagesthewebconfConference Proceedingsconference-collections
short-paper
Open access

Is the 'Impression Log' Beneficial to Evaluating News Recommender Systems? No, it is Not!

Published: 13 May 2024 Publication History

Abstract

This paper aims to answer the question of whether to use the impression log in evaluating news recommendation models. We start with a claim that the testing with the impression log composed of only hard-negative news (i.e., impression (IMP)-based test) is not beneficial to evaluating the models precisely. Based on the claim, we discuss a way of evaluating models by employing all kinds of negative news articles (i.e., Total test). Also, we propose a more-efficient way of evaluating models by sampling only a small number of negative articles (i.e., random-sampling (RS)-based test). We verify our claim by extensively comparing the evaluation results on six models from the IMP-based, Total, and RS-based tests: the RS-based test shows more accurate results than the IMP-based test in determining the superiority among the models while providing higher efficiency than the Total test. Therefore, our answer to the question above would be "do not employ the impression log in testing models even if it is available." This result is quite meaningful since it enables news recommendation researchers and practitioners, who have been using the impression log thus going to the wrong way, to turn to the right one.

Supplemental Material

MP4 File
Supplemental video

References

[1]
J Ahn et al. 2023. Is the Impression Log Beneficial to Effective Model Training in News Recommender Systems? No, It's NOT. In WWW. 61--64.
[2]
M An et al. 2019. Neural News Recommendation with Long- and Short-term User Representations. In ACL. 336--345.
[3]
HK Bae et al. 2023. A competition-aware approach to accurate TV show recommendation. In IEEE ICDE. IEEE, 2822--2834.
[4]
HK Bae et al. 2023. LANCER: A Lifetime-aware News Recommender System. In AAAI, Vol. 37. 4141--4148.
[5]
J Kim et al. 2023. Read between the interactions: Understanding non-interacted items for accurate multimedia recommendation. Computer Science and Information Systems 20, 3 (2023), 933--948.
[6]
T Kim et al. 2022. Is it enough just looking at the title? Leveraging body text to enrich title words towards accurate news recommendation. In ACM CIKM. 4138--4142.
[7]
T Kim et al. 2022. MARIO: modality-aware attention and modality-preserving decoders for multimedia recommendation. In ACM CIKM. 993--1002.
[8]
W Krichene et al. 2020. On Sampled Metrics for Item Recommendation. In ACM KDD. 1748--1757.
[9]
H Lim et al. 2022. AiRS: A Large-Scale Recommender System at NAVER News. In IEEE ICDE. 3386--3398.
[10]
Z Mao et al. 2021. Neural News Recommendation with Collaborative News Encoding and Structural User Encoding. In EMNLP Findings. 46--55.
[11]
Z Mao et al. 2022. DIGAT: Modeling News Recommendation with Dual-Graph Interaction. In EMNLP Findings. 6595--6607.
[12]
C Wu et al. 2019. Neural News Recommendation with Attentive Multi-View Learning. In IJCAI. 3863--3869.
[13]
CWu et al. 2019. Neural News Recommendation with Multi-Head Self-Attention. In EMNLP-IJCNLP. 6389--6394.
[14]
C Wu et al. 2019. NPA: Neural News Recommendation with Personalized Attention. In ACM KDD. 2576--2584.
[15]
F Wu et al. 2020. MIND: A Large-scale Dataset for News Recommendation. In ACL. 3597--3606.

Index Terms

  1. Is the 'Impression Log' Beneficial to Evaluating News Recommender Systems? No, it is Not!

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    WWW '24: Companion Proceedings of the ACM Web Conference 2024
    May 2024
    1928 pages
    ISBN:9798400701726
    DOI:10.1145/3589335
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 13 May 2024

    Check for updates

    Author Tags

    1. deep neural networks
    2. model evaluation
    3. news recommender systems

    Qualifiers

    • Short-paper

    Funding Sources

    • RS-2022-00155586
    • 2022-0-00352
    • 2020-0-01373

    Conference

    WWW '24
    Sponsor:
    WWW '24: The ACM Web Conference 2024
    May 13 - 17, 2024
    Singapore, Singapore

    Acceptance Rates

    Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 298
      Total Downloads
    • Downloads (Last 12 months)298
    • Downloads (Last 6 weeks)36
    Reflects downloads up to 07 Mar 2025

    Other Metrics

    Citations

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media