ABSTRACT
Ranking algorithms as an essential component of retrieval systems have been constantly improved in previous studies, especially regarding relevance-based utilities. In recent years, more and more research attempts have been proposed regarding fairness in rankings due to increasing concerns about potential discrimination and the issue of echo chamber. These attempts include traditional score-based methods that allocate exposure resources to different groups using pre-defined scoring functions or selection strategies and learning-based methods that learn the scoring functions based on data samples. Learning-based models are more flexible and achieve better performance than traditional methods. However, most of the learning-based models were trained and tested on outdated datasets where fairness labels are barely available. State-of-art models utilize relevance-based utility scores as a substitute for the fairness labels to train their fairness-aware loss, where plugging in the substitution does not guarantee the minimum loss. This inconsistency challenges the model's accuracy and performance, especially when learning is achieved by gradient descent. Hence, we propose a distribution-based fair learning framework (DLF) that does not require labels by replacing the unavailable fairness labels with target fairness exposure distributions. Experimental studies on TREC fair ranking track dataset confirm that our proposed framework achieves better fairness performance while maintaining better control over the fairness-relevance trade-off than state-of-art fair ranking frameworks.
- Gabriel Bénédict Ali Vardasbi, Shashank Gupta, Maria Heuss, Pooya Khandel, Ming Li, and Fatemeh Sarvi. 2021. The University of Amsterdam at the TREC 2021 Fair Ranking Track. In Proceedings of TREC.Google Scholar
- Christopher JC Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning, Vol. 11, 23--581 (2010), 81.Google Scholar
- Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning. 129--136.Google ScholarDigital Library
- Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. 335--336.Google ScholarDigital Library
- L Elisa Celis, Sayash Kapoor, Farnood Salehi, and Nisheeth Vishnoi. 2019. Controlling polarization in personalization: An algorithmic framework. In Proceedings of the conference on fairness, accountability, and transparency. 160--169.Google ScholarDigital Library
- L Elisa Celis, Damian Straszak, and Nisheeth K Vishnoi. 2017. Ranking with fairness constraints. arXiv preprint arXiv:1704.06840 (2017).Google Scholar
- Fumian Chen and Hui Fang. 2022. An Exploration of Learning-to-re-rank Using a Two-step Framework for Fair Ranking. In Proceedings of TREC.Google Scholar
- Sachin Pathiyan Cherumanal, Marwah Alaofi, Reham Abdullah Altalhi, Elham Naghizade, Falk Scholer, and Damiano Spina. 2022. RMIT CIDDA IR at the TREC 2022 Fair Ranking Track. In Proceedings of TREC.Google Scholar
- Sachin Pathiyan Cherumanal, Damiano Spina, Falk Scholer, and W Bruce Croft. 2021. RMIT at TREC 2021 Fair Ranking Track. In Proceedings of TREC.Google Scholar
- Fernando Diaz, Bhaskar Mitra, Michael D Ekstrand, Asia J Biega, and Ben Carterette. 2020. Evaluating stochastic rankings with expected exposure. In Proceedings of the 29th ACM international conference on information & knowledge management. 275--284.Google ScholarDigital Library
- Michael D Ekstrand, Graham McDonald, Amifa Raj, and Isaac Johnson. 2023. Overview of the TREC 2022 Fair Ranking Track. arXiv preprint arXiv:2302.05558 (2023).Google Scholar
- Ruoyuan Gao, Yingqiang Ge, and Chirag Shah. 2022. FAIR: Fairness-aware information retrieval evaluation. Journal of the Association for Information Science and Technology (2022).Google Scholar
- Ruoyuan Gao and Chirag Shah. 2021. Addressing bias and fairness in search systems. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 2643--2646.Google ScholarDigital Library
- Thomas Jaenich, Graham McDonald, and Iadh Ounis. 2021. University of Glasgow Terrier Team at the TREC 2022 Fair Ranking Track. In Proceedings of TREC.Google Scholar
- Preethi Lahoti, Krishna P Gummadi, and Gerhard Weikum. 2019. ifair: Learning individually fair data representations for algorithmic decision making. In 2019 ieee 35th international conference on data engineering (icde). IEEE, 1334--1345.Google Scholar
- Tai Le Quy, Arjun Roy, Vasileios Iosifidis, Wenbin Zhang, and Eirini Ntoutsi. 2022. A survey on datasets for fairness-aware machine learning. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery (2022), e1452.Google Scholar
- Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021). 2356--2362.Google ScholarDigital Library
- Yuanhua Lv and ChengXiang Zhai. 2009. A comparative study of methods for estimating query language models with pseudo feedback. In Proceedings of the 18th ACM conference on Information and knowledge management. 1895--1898.Google ScholarDigital Library
- MilkaLichtblau. [n.,d.]. DELTR-experiments/Data/Engineeringstudents at master · Milkalichtblau/DELTR-Experiments. https://github.com/MilkaLichtblau/DELTR-Experiments/tree/master/data/EngineeringStudents/Google Scholar
- Gourab K Patro, Lorenzo Porcaro, Laura Mitchell, Qiuyue Zhang, Meike Zehlike, and Nikhil Garg. 2022. Fair ranking: a critical review, challenges, and future directions. arXiv preprint arXiv:2201.12662 (2022).Google Scholar
- ProPublica. [n.,d.]. Compas recidivism risk score data and analysis. https://www.propublica.org/datastore/dataset/compas-recidivism-risk-score-data-and-analysisGoogle Scholar
- Amifa Raj, Connor Wood, Ananda Montoly, and Michael D Ekstrand. 2020. Comparing fair ranking metrics. arXiv preprint arXiv:2009.01311 (2020).Google Scholar
- Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019).Google Scholar
- Stephen Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1993. Okapi at TREC. National Institute of Standards and Technology, 109--123.Google Scholar
- Tetsuya Sakai, Jin Young Kim, and Inho Kang. 2022. A Versatile Framework for Evaluating Ranked Lists in terms of Group Fairness and Relevance. arXiv preprint arXiv:2204.00280 (2022).Google Scholar
- Piotr Sapiezynski, Wesley Zeng, Ronald E Robertson, Alan Mislove, and Christo Wilson. 2019. Quantifying the impact of user attentionon fair group representation in ranked lists. In Companion proceedings of the 2019 world wide web conference. 553--562.Google ScholarDigital Library
- Julia Stoyanovich, Ke Yang, and HV Jagadish. 2018. Online set selection with fairness and diversity constraints. In Proceedings of the EDBT Conference.Google Scholar
- Xiao Wang, Craig Macdonald, Nicola Tonellotto, and Iadh Ounis. 2023. ColBERT-PRF: Semantic pseudo-relevance feedback for dense passage and document retrieval. ACM Transactions on the Web, Vol. 17, 1 (2023), 1--39.Google ScholarDigital Library
- Yuan Wang, Zhiqiang Tao, and Yi Fang. 2022. A Meta-learning Approach to Fair Ranking. (2022).Google Scholar
- Yuyan Wang, Xuezhi Wang, Alex Beutel, Flavien Prost, Jilin Chen, and Ed H Chi. 2021. Understanding and improving fairness-accuracy trade-offs in multi-task learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1748--1757.Google ScholarDigital Library
- Ke Yang, Vasilis Gkatzelis, and Julia Stoyanovich. 2019. Balanced ranking with diversity constraints. arXiv preprint arXiv:1906.01747 (2019).Google Scholar
- Ke Yang and Julia Stoyanovich. 2017. Measuring fairness in ranked outputs. In Proceedings of the 29th international conference on scientific and statistical database management. 1--6.Google ScholarDigital Library
- Meike Zehlike, Francesco Bonchi, Carlos Castillo, Sara Hajian, Mohamed Megahed, and Ricardo Baeza-Yates. 2017. Fa* ir: A fair top-k ranking algorithm. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 1569--1578.Google ScholarDigital Library
- Meike Zehlike and Carlos Castillo. 2020. Reducing disparate exposure in ranking: A learning to rank approach. In Proceedings of The Web Conference 2020. 2849--2855.Google ScholarDigital Library
- Meike Zehlike, Philipp Hacker, and Emil Wiedemann. 2020. Matching code and law: achieving algorithmic fairness with optimal transport. Data Mining and Knowledge Discovery, Vol. 34, 1 (2020), 163--200.Google ScholarDigital Library
- Meike Zehlike, Tom Sühr, Ricardo Baeza-Yates, Francesco Bonchi, Carlos Castillo, and Sara Hajian. 2022. Fair Top-k Ranking with multiple protected groups. Information Processing & Management, Vol. 59, 1 (2022), 102707.Google ScholarDigital Library
- Meike Zehlike, Ke Yang, and Julia Stoyanovich. 2021. Fairness in ranking: A survey. arXiv preprint arXiv:2103.14000 (2021).Google Scholar
- Jin Zhuoqi, Hideo Joho, and Sumio Fujita. 2021. TKB48 at TREC 2021 Fairness Ranking Track. In Proceedings of TREC.Google Scholar
Index Terms
- Learn to be Fair without Labels: A Distribution-based Learning Framework for Fair Ranking
Recommendations
Search results diversification for effective fair ranking in academic search
AbstractProviding users with relevant search results has been the primary focus of information retrieval research. However, focusing on relevance alone can lead to undesirable side effects. For example, small differences between the relevance scores of ...
Marginal-Certainty-Aware Fair Ranking Algorithm
WSDM '23: Proceedings of the Sixteenth ACM International Conference on Web Search and Data MiningRanking systems are ubiquitous in modern Internet services, including online marketplaces, social media, and search engines. Traditionally, ranking systems only focus on how to get better relevance estimation. When relevance estimation is available, they ...
T2-fair: a two-tiered time and throughput fair scheduler for multi-rate WLANs
MSWiM '06: Proceedings of the 9th ACM international symposium on Modeling analysis and simulation of wireless and mobile systemsLow throughput due to unfairness is a key problem in multi-rate wireless local area networks. To promote fairness and hence throughput, T2-Fair groups flows according to their average data rate, provides each group fair time allocations and ensures ...
Comments