Empirical Evaluation of Predictive Models: A keynote at ECIR 2022
Article No.: 2, Pages 1 - 5
Abstract
I give a brief overview of my recent keynote at the 2022 European Conference on Information Retrieval that was held in Stavanger, Norway. I pay particular attention to some basic questions involving the F-score that appear to lead to confusion. I also settle a question raised at the conference by reconstructing an account from Van Rijsbergen's classic text Information Retrieval.
References
[1]
Yu Chen, Telmo Silva Filho, Ricardo Prudencio, Tom Diethe, and Peter Flach. β3-IRT: A new item response model and its applications. In 22nd International Conference on Artificial Intelligence and Statistics, pages 1013--1021, 2019. URL https://proceedings.mlr.press/v89/chen19b.html.
[2]
Peter Flach. Performance evaluation in machine learning: the good, the bad, the ugly, and the way forward. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9808--9814, 2019. URL https://ojs.aaai.org//index.php/AAAI/article/view/5055.
[3]
Peter Flach and Meelis Kull. Precision-recall-gain curves: PR analysis done right. In Advances in Neural Information Processing Systems, pages 838--846, 2015. URL http://people.cs.bris.ac.uk/~flach/PRGcurves/.
[4]
José Hernández-Orallo, Peter Flach, and Cèsar Ferri. A unified view of performance metrics: translating threshold choice into expected classification loss. Journal of Machine Learning Research, 13:2813--2869, 2012. URL https://www.jmlr.org/papers/v13/hernandez-orallo12a.html.
[5]
Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos-Rodriguez, Meelis Kull, and Peter Flach. Classifier calibration: How to assess and improve predicted class probabilities: a survey. arXiv preprint arXiv:2112.10327, 2021. URL https://arxiv.org/abs/2112.10327.
[6]
Hao Song and Peter Flach. Efficient and robust model benchmarks with item response theory and adaptive testing. International Journal of Interactive Multimedia & Artificial Intelligence, 6(5), 2021. URL https://www.ijimai.org/journal/bibcite/reference/2901.
Recommendations
Predictive Models in Personalized Medicine: Neural Information Processing Systems (NIPS), 2010 workshop report
This workshop report is an overview of the Predictive Models in Personalized Medicine workshop held on Dec. 11, 2010 at 2010 Neural Information Processing Systems (NIPS) Conference in Whistler, Canada. The workshop included 3 keynote talks and 6 oral ...
Comments
Information & Contributors
Information
Published In
![cover image ACM SIGIR Forum](/cms/asset/b4bae45f-4236-4b12-ae35-1bb6af0ee9c3/3308774.cover.jpg)
Copyright © 2023 Copyright is held by the owner/author(s).
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 January 2023
Published in SIGIR Volume 56, Issue 1
Check for updates
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 25Total Downloads
- Downloads (Last 12 months)6
- Downloads (Last 6 weeks)0
Reflects downloads up to 13 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in