Abstract
In Information Retrieval (IR), evaluation metrics continuously play an important role. Recently, some risk measures have been proposed to evaluate the downside performance or the performance variance of an assumingly advanced IR method in comparison with a baseline method. In this paper, we propose a novel risk metric, by applying the Value at Risk theory (VaR, which has been widely used in financial investment) to IR risk evaluation. The proposed metric (VaR_IR) is implemented in the light of typical IR effectiveness metrics (e.g. AP) and used to evaluate the participating systems submitted to Session Tracks and compared with other risk metrics. The empirical evaluation has shown that VaR_IR is complementary to and can be integrated with the effectiveness metrics to provide a more comprehensive evaluation method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amati, G., Carpineto, C., Romano, G.: Query difficulty, robustness, and selective application of query expansion. In: McDonald, S., Tait, J. (eds.) ECIR 2004. LNCS, vol. 2997, pp. 127–137. Springer, Heidelberg (2004). doi:10.1007/978-3-540-24752-4_10
Collins-Thompson, K.: Reducing the risk of query expansion via robust constrained optimization. In: Proceedings of CIKM 2009, pp. 837–846. ACM (2009)
Collins-Thompson, K., Macdonald, C., Bennett, P., Diaz, F., Voorhees, E.M.: Trec 2014 web track overview. Technical report, DTIC Document (2015)
Linsmeier, T.J., Pearson, N.D.: Value at risk. Financ. Anal. J. 56(2), 47–67 (2000)
Morgan, J.: Riskmetrics: Technical Document. Morgan Guaranty Trust Company of New York, New York (1996)
Robertson, S.E.: The probability ranking principle in IR. J. Documentation 33, 294–304 (1977)
van Rijsbergen, C.J.: Information Retrieval. Butterworths, Oxford (1979)
Zhai, C., Lafferty, J.D.: A risk minimization framework for information retrieval. Inf. Process. Manag. 42(1), 31–55 (2006)
Zhang, P., Song, D., Wang, J., Hou, Y.: Bias-variance decomposition of IR evaluation. In: Proceedings of SIGIR, vol. 2013, pp. 1021–1024 (2013)
Acknowledgements
The work presented in this paper is sponsored in part by the Chinese National Program on Key Basic Research Project (973 Program, grant No. 2013CB329304, 2014CB744604), the Chinese 863 Program (grant No. 2015AA015403), the Natural Science Foundation of China (grant No. 61402324, 61272265), and the Research Fund for the Doctoral Program of Higher Education of China (grant No. 20130032120044).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Wang, M., Zhang, P., Song, D., Wang, J. (2016). Value at Risk for Risk Evaluation in Information Retrieval. In: Lin, CY., Xue, N., Zhao, D., Huang, X., Feng, Y. (eds) Natural Language Understanding and Intelligent Applications. ICCPOL NLPCC 2016 2016. Lecture Notes in Computer Science(), vol 10102. Springer, Cham. https://doi.org/10.1007/978-3-319-50496-4_56
Download citation
DOI: https://doi.org/10.1007/978-3-319-50496-4_56
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-50495-7
Online ISBN: 978-3-319-50496-4
eBook Packages: Computer ScienceComputer Science (R0)