skip to main content
research-article

Empirical Evaluation of Predictive Models: A keynote at ECIR 2022

Published: 27 January 2023 Publication History

Abstract

I give a brief overview of my recent keynote at the 2022 European Conference on Information Retrieval that was held in Stavanger, Norway. I pay particular attention to some basic questions involving the F-score that appear to lead to confusion. I also settle a question raised at the conference by reconstructing an account from Van Rijsbergen's classic text Information Retrieval.

References

[1]
Yu Chen, Telmo Silva Filho, Ricardo Prudencio, Tom Diethe, and Peter Flach. β3-IRT: A new item response model and its applications. In 22nd International Conference on Artificial Intelligence and Statistics, pages 1013--1021, 2019. URL https://proceedings.mlr.press/v89/chen19b.html.
[2]
Peter Flach. Performance evaluation in machine learning: the good, the bad, the ugly, and the way forward. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9808--9814, 2019. URL https://ojs.aaai.org//index.php/AAAI/article/view/5055.
[3]
Peter Flach and Meelis Kull. Precision-recall-gain curves: PR analysis done right. In Advances in Neural Information Processing Systems, pages 838--846, 2015. URL http://people.cs.bris.ac.uk/~flach/PRGcurves/.
[4]
José Hernández-Orallo, Peter Flach, and Cèsar Ferri. A unified view of performance metrics: translating threshold choice into expected classification loss. Journal of Machine Learning Research, 13:2813--2869, 2012. URL https://www.jmlr.org/papers/v13/hernandez-orallo12a.html.
[5]
Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos-Rodriguez, Meelis Kull, and Peter Flach. Classifier calibration: How to assess and improve predicted class probabilities: a survey. arXiv preprint arXiv:2112.10327, 2021. URL https://arxiv.org/abs/2112.10327.
[6]
Hao Song and Peter Flach. Efficient and robust model benchmarks with item response theory and adaptive testing. International Journal of Interactive Multimedia & Artificial Intelligence, 6(5), 2021. URL https://www.ijimai.org/journal/bibcite/reference/2901.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM SIGIR Forum
ACM SIGIR Forum  Volume 56, Issue 1
June 2022
109 pages
ISSN:0163-5840
DOI:10.1145/3582524
Issue’s Table of Contents
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 January 2023
Published in SIGIR Volume 56, Issue 1

Check for updates

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 25
    Total Downloads
  • Downloads (Last 12 months)6
  • Downloads (Last 6 weeks)0
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media