Emine Yilmaz
Stephen Robertson
ABSTRACT
In the field of information retrieval, one is often faced with the problem of computing the correlation between two ranked lists. The most commonly used statistic that quantifies this correlation is Kendall's Τ. Often times, in the information retrieval community, discrepancies among those items having high rankings are more important than those among items having low rankings. The Kendall's Τ statistic, however, does not make such distinctions and equally penalizes errors both at high and low rankings.
In this paper, we propose a new rank correlation coefficient, AP correlation (Τap), that is based on average precision and has a probabilistic interpretation. We show that the proposed statistic gives more weight to the errors at high rankings and has nice mathematical properties which make it easy to interpret. We further validate the applicability of the statistic using experimental data.
- J. A. Aslam, V. Pavlu, and R. Savell. A unified model for metasearch, pooling, and system evaluation. In O. Frieder, J. Hammer, S. Quershi, and L. Seligman, editors, Proceedings of the Twelfth International Conference on Information and Knowledge Management, pages 484--491. ACM Press, November 2003. Google Scholar
Digital Library
- C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 25--32, New York, NY, USA, 2004. ACM Press. Google ScholarDigital Library
- B. Carterette and J. Allan. Incremental test collections. In CIKM '05: Proceedings of the 14th ACM international conference on Information and knowledge management, pages 680--687, New York, NY, USA, 2005. ACM Press. Google ScholarDigital Library
- T. Cover and J. Thomas. Elements of Information Theory. Wiley, 1991. Google ScholarDigital Library
- R. Fagin, R. Kumar, and D. Sivakumar. Comparing top k lists. In SODA '03: Proceedings of the fourteenth annual ACM-SIAM symposium on Discrete algorithms, pages 28--36, Philadelphia, PA, USA, 2003. Society for Industrial and Applied Mathematics. Google ScholarDigital Library
- T. H. Haveliwala, A. Gionis, D. Klein, and P. Indyk. Evaluating strategies for similarity search on the web. In WWW '02: Proceedings of the 11th international conference on World Wide Web, pages 432--442, New York, NY, USA, 2002. ACM. Google ScholarDigital Library
- M. Kendall. A new measure of rank correlation. Biometrica, 30(1-2):81--89, 1938.Google ScholarCross Ref
- M. Melucci. On rank correlation in information retrieval evaluation. SIGIR Forum, 41(1):18--33, 2007. Google ScholarDigital Library
- M. Sanderson and H. Joho. Forming test collections with no system pooling. In SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 33--40, New York, NY, USA, 2004. ACM. Google ScholarDigital Library
- M. Sanderson and I. Soboroff. Problems with kendall's tau. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 839--840, New York, NY, USA, 2007. ACM. Google ScholarDigital Library
- G. S. Shieh. A weighted kendall's tau statistic. Statistics & Probability Letters, 39:17--24, 1998.Google ScholarCross Ref
- I. Soboroff, C. Nicholas, and P. Cahan. Ranking retrieval systems without relevance judgments. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 66--73, New Orleans, Louisiana, USA, Sept. 2001. ACM Press, New York. Google ScholarDigital Library
- E. M. Voorhees. Evaluation by highly relevant documents. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 74--82. ACM Press, 2001. Google ScholarDigital Library
- E. M. Voorhees. Overview of the TREC 2004 robust retrieval track. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), 2004.Google Scholar
- D. D. Wackerly, W. Mendenhall, and R. L. Scheaffer. Mathematical Statistics with Applications. Duxbury Advanced Series, 2002.Google Scholar
- S. Wu and F. Crestani. Methods for ranking information retrieval systems without relevance judgments. In SAC '03: Proceedings of the 2003 ACM symposium on Applied computing, pages 811--816, 2003. Google ScholarDigital Library
- E. Yilmaz and J. A. Aslam. Estimating average precision with incomplete and imperfect judgments. In Proceedings of the Fifteenth ACM International Conference on Information and Knowledge Management. ACM Press, November 2006. Google ScholarDigital Library
Index Terms
- A new rank correlation coefficient for information retrieval
Recommendations
On rank correlation and the distance between rankings
SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrievalRank correlation statistics are useful for determining whether a there is a correspondence between two measurements, particularly when the measures themselves are of less interest than their relative ordering. Kendall's - in particular has found use in ...
The Treatment of Ties in AP Correlation
ICTIR '17: Proceedings of the ACM SIGIR International Conference on Theory of Information RetrievalThe Kendall tau and AP correlation coefficients are very commonly use to compare two rankings over the same set of items. Even though Kendall tau was originally defined assuming that there are no ties in the rankings, two alternative versions were soon ...
Order Statistics Correlation Coefficient as a Novel Association Measurement With Applications to Biosignal Analysis
In this paper, we propose a novel correlation coefficient based on order statistics and rearrangement inequality. The proposed coefficient represents a compromise between the Pearson's linear coefficient and the two rank-based coefficients, namely ...
Comments