Evaluating Query Performance Predictors Based on Brownian Distance Correlation

Abstract

Modern information retrieval system suffers from radical variance performance even though its mean performance is well. In order to solve this problem, the Query Performance Predictor, predicting the performance of query without relevance assessment information , which is quite expensive or even infeasible in real system, was studied in the past decade. To evaluate the Query Performance Predictor, the correlation coefficient between predictor’s output and the Average Precision of query is calculated. Current works mainly employ correlation metrics including the Pearson correlation coefficient, Spearman’s Rho and Kendall’s tau. However, these correlation metrics have some limitation in evaluating the quality of predictor. To this end, we introduce a novel metric based on Brownian Distance Correlation (Dcor) in evaluating query performance predictor, which is able to measure the independence relationship apart from the linear correlation relationship between two variables. Therefore, the new method can report more reliable results especially when there are nonlinear or non-monotone relationships. We conduct a series of experiments on several standard TREC datasets and compare the results between Dcor and the classic metrics. In the experiments, the novel metric exhibits consistent evaluating results compared with the three classic coefficients. However, the results suggest a better stability when tuning the predictor’s parameters.