ColBERT-FairPRF: Towards Fair Pseudo-Relevance Feedback in Dense Retrieval

Abstract

Pseudo-relevance feedback mechanisms have been shown to be useful in improving the effectiveness of search systems for retrieving the most relevant items in response to a user’s query. However, there has been little work investigating the relationship between pseudo-relevance feedback and fairness in ranking. Indeed, using the feedback from an initial retrieval to revise a query can in principle also allow to optimise objectives beyond relevance, such as the fairness of the search results. In this work, we show how a feedback mechanism based on the successful ColBERT-PRF model can be used for retrieving fairer search results. Therefore, we propose a novel fair feedback mechanism for multiple representation dense retrieval (ColBERT-FairPRF), which enhances the distribution of exposure over groups of documents in the search results by fairly extracting the feedback embeddings that are added to the user’s query representation. To fairly extract representative embeddings, we apply a clustering approach since traditional methods based on counting are not applicable in the dense retrieval space. Our results on the 2021 TREC Fair Ranking Track test collection demonstrate the effectiveness of our method compared to ColBERT-PRF, with statistical significant improvements of up to $\sim$19% in Attention Weighted Ranked Fairness. To the best of our knowledge, ColBERT-FairPRF is the first query expansion method for fairness in multiple representation dense retrieval.