abstract

Fair Ranking Metrics

Author:

Amifa RajAuthors Info & Claims

RecSys '22: Proceedings of the 16th ACM Conference on Recommender Systems

Pages 742 - 743

https://doi.org/10.1145/3523227.3547430

Published: 13 September 2022 Publication History

Get Access

Abstract

Information access systems such as search engines and recommender systems often display results in a sorted ranked list based on their relevance. Fairness of these ranked list has received attention as an important evaluation criteria along with traditional metrics such as utility or accuracy. Fairness broadly involves both provider and consumer side fairness at both group and individual levels. Several fair ranking metrics have been proposed to measure group fairness for providers based on various “sensitive attributes”. These metrics differ in their fairness goal, assumptions, and implementations. Although there are several fair ranking metrics to measure group fairness, multiple open challenges still exist in this area to consider.

In my thesis, I work on the area of fair ranking metrics for provider-side group fairness. I am interested in understanding the fairness concepts and practical applications of these metrics to identify their strength and limitations to aid the researchers and practitioner by pointing out the gaps. Moreover, I will contribute to this research area by focusing on some of the limitations like considering different browsing models and bias in relevance information.

References

[1]

Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, and Ben Carterette. 2020. Evaluating Stochastic Rankings with Expected Exposure. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (Virtual Event, Ireland) (CIKM ’20). Association for Computing Machinery, New York, NY, USA, 275–284. https://doi.org/10.1145/3340531.3411962

Digital Library

Google Scholar

[2]

Amifa Raj and Michael D Ekstrand. 2022. Measuring Fairness in Ranked Results: An Analytical and Empirical Comparison. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 726–736.

Digital Library

Google Scholar

[3]

Ashudeep Singh and Thorsten Joachims. 2018. Fairness of Exposure in Rankings. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (London, United Kingdom) (KDD ’18). Association for Computing Machinery, New York, NY, USA, 2219–2228. https://doi.org/10.1145/3219819.3220088

Digital Library

Google Scholar

[4]

Yisong Yue, Rajan Patel, and Hein Roehrig. 2010. Beyond Position Bias: Examining Result Attractiveness as a Source of Presentation Bias in Clickthrough Data. In Proceedings of the 19th International Conference on World Wide Web (Raleigh, North Carolina, USA) (WWW ’10). Association for Computing Machinery, New York, NY, USA, 1011–1018. https://doi.org/10.1145/1772690.1772793

Digital Library

Google Scholar

Cited By

View all

Vente TEkstrand MBeel J(2023)Introducing LensKit-Auto, an Experimental Automated Recommender System (AutoRecSys) ToolkitProceedings of the 17th ACM Conference on Recommender Systems10.1145/3604915.3610656(1212-1216)Online publication date: 14-Sep-2023
https://dl.acm.org/doi/10.1145/3604915.3610656

Index Terms

Fair Ranking Metrics
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results

Index terms have been assigned to the content through auto-classification.

Recommendations

Are Slice-Based Cohesion Metrics Actually Useful in Effort-Aware Post-Release Fault-Proneness Prediction? An Empirical Study
Background. Slice-based cohesion metrics leverage program slices with respect to the output variables of a module to quantify the strength of functional relatedness of the elements within the module. Although slice-based cohesion metrics have been ...
Correlation, Prediction and Ranking of Evaluation Metrics in Information Retrieval
Advances in Information Retrieval
Abstract
Given limited time and space, IR studies often report few evaluation metrics which must be carefully selected. To inform such selection, we first quantify correlation between 23 popular IR metrics on 8 TREC test collections. Next, we investigate ...
Metrics for class cohesion and similarity between methods
ACMSE '06: Proceedings of the 44th annual ACM Southeast Conference

Class cohesion is one of the desirable properties in object oriented designs. But, designers and managers need a good metric for this property to help them evaluate, compare and choose among various possible solutions to a given problem. In this paper, ...

Comments

Information & Contributors

Information

Published In

RecSys '22: Proceedings of the 16th ACM Conference on Recommender Systems

September 2022

743 pages

ISBN:9781450392785

DOI:10.1145/3523227

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 September 2022

Check for updates

Author Tags

Qualifiers

Abstract
Research
Refereed limited

Conference

RecSys '22

Sponsor:

RecSys '22: Sixteenth ACM Conference on Recommender Systems

September 18 - 23, 2022

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
221
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)5

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Vente TEkstrand MBeel J(2023)Introducing LensKit-Auto, an Experimental Automated Recommender System (AutoRecSys) ToolkitProceedings of the 17th ACM Conference on Recommender Systems10.1145/3604915.3610656(1212-1216)Online publication date: 14-Sep-2023
https://dl.acm.org/doi/10.1145/3604915.3610656

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Index Terms

Recommendations

Are Slice-Based Cohesion Metrics Actually Useful in Effort-Aware Post-Release Fault-Proneness Prediction? An Empirical Study

Correlation, Prediction and Ranking of Evaluation Metrics in Information Retrieval

Metrics for class cohesion and similarity between methods

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Share

Share this Publication link

Share on social media

Affiliations