Direct measurement of training query quality for learning to rank

The conventional application of learning to rank algorithms tends to use as many training queries as possible to leverage the benefit brought by a large amount of labeled data. However, the use of all training queries available may also include the low quality ones, and consequently, degrades the retrieval effectiveness, hence the need for selecting training queries. Existing training query selection approaches incorporate a variety of indirect indicators of the training queries such as the query performance predictors and the relevance scores into a classification or regression based approach. In this paper, we propose to select training queries by the direct measurement of the training query quality, namely the resulting retrieval performance on a subset of validation queries, instead of the indirect indicators that may not have strong correlations with a training query's quality. Evaluation on the standard LETOR 4.0 dataset shows that our proposed approach outperforms the state-of-the-art baselines.