Wenbin Cai
ABSTRACT
Active learning for ranking, which is to selectively label the most informative examples, has been widely studied in recent years. In this paper, we propose a general active learning for ranking strategy called Variance Maximization (VM). The algorithm relies on noise injection to perturb the original unlabeled examples and generate the rank distribution of each example. Using a DCG-like gain function to measure each ranked list sampled from the rank distribution, Variance Maximization selects the unlabeled example with the largest variance in the gain. The VM strategy is applied at both the query level and the document level, and a two-stage active learning algorithm is further derived. Experimental results on both the LETOR 4.0 dataset and a real-world Web search ranking dataset have demonstrated the effectiveness of the proposed active learning approach.
- P. Cai, W. Gao, A. Zhou, and K. F. Wong. Relevant knowledge helps in choosing right teacher: Active query selection for ranking adaptation. In Proceedings of SIGIR '11, pages 115--124, 2011. Google Scholar
Digital Library
- P. Donmez and J. G. Carbonell. Optimizing estimated loss reduction for active sampling in rank learning. In Proceedings of ICML '08, pages 248--255, 2008. Google ScholarDigital Library
- Y. Freund, R. Iyer, R. E. Schapire, and Y. Singer. An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, pages 933--969, 2003. Google ScholarDigital Library
- Y. Freund, H. S. Seung, E. Shamir, and N. Tishby. Selective sampling using the query by committee algorithm. Machine Learning, pages 133--168, 1997. Google ScholarDigital Library
- J. Friedman. Greedy function approximation: a gradient boosting machine. Annals of Statistics, pages 1189--1232, 2001.Google ScholarCross Ref
- R. Herbrich, T. Graepel, and K. Obermayer. Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers. MIT Press, 2000.Google Scholar
- K. Jarvelin and J. Kekalainen. IR evaluation methods for retrieving highly relevant documents. In Proceedings of SIGIR '00, pages 41--48, 2000. Google ScholarDigital Library
- B. Long, O. Chapelle, Y. Zhang, Y. Chang, Z. Zheng, and B. Tseng. Active learning for ranking through expected loss optimization. In Proceedings of SIGIR '10, pages 267--274, 2010. Google ScholarDigital Library
- B. Settles. Active learning literature survey. Computer sciences technical report, University of Wisconsin-Madison, 2009.Google Scholar
- R. Silva, M. A. Gonçalves, and A. Veloso. Rule-based active sampling for learning to rank. In Proceedings of ECML PKDD '11, pages 240--255, 2011. Google ScholarDigital Library
- M. Taylor, J. Guiver, S. Robertson, and T. Minka. SoftRank: Optimizing non-smooth rank metrics. In Proceedings of WSDM '08, pages 77--86, 2008. Google ScholarDigital Library
- E. Yilmaz and S. Robertson. Deep versus shallow judgments in learning to rank. In Proceedings of SIGIR '09, pages 662--663, 2009. Google ScholarDigital Library
- H. Yu. SVM selective sampling for ranking with application to data retrieval. In Proceedings of KDD '05, pages 354--363, 2005. Google ScholarDigital Library
Index Terms
- Variance maximization via noise injection for active sampling in learning to rank
Recommendations
A variance maximization criterion for active learning
A novel active learning approach is proposed, which measures the value of unlabeled data by its predictive variance.To measure the informativeness and representativeness of unlabeled instances, two types of variance are introduced.Excellent performance ...
Active Learning for Web Search Ranking via Noise Injection
Learning to rank has become increasingly important for many information retrieval applications. To reduce the labeling cost at training data preparation, many active sampling algorithms have been proposed. In this article, we propose a novel active ...
Active Learning for Efficient Partial Improvement of Learning to Rank
From Born-Physical to Born-Virtual: Augmenting Intelligence in Digital LibrariesAbstractIn this study, we propose a framework that allows partial improvement of ranking algorithms based on feedback from search engine administrators, where partial improvement refers to changing the rank of a particular document for a particular type ...
Comments