Abstract
Evaluation has a crucial role in (IR) since it allows for identifying possible points of failure of an IR approach, thus addressing them to improve its effectiveness. Developing tools to support researchers and analysts when analyzing results and investigating strategies to improve IR system performance can help make the analysis easier and more effective. In this paper we discuss a VA-based approach to support the analyst when deciding whether or not to investigate re-ranking to improve the system effectiveness measured after a retrieval run. Our approach is based on effectiveness measures that exploit graded relevance judgements and it provides both a principled and intuitive way to support analysis. A prototype is described and exploited to discuss some case studies based on TREC data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Harman, D., Buckley, C.: Overview of the Reliable Information Access Workshop. Information Retrieval 12, 615–641 (2009)
Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information System 20, 422–446 (2002)
Keskustalo, H., Järvelin, K., Pirkola, A., Kekäläinen, J.: Intuition-supporting visualization of user’s performance based on explicit negative higher-order relevance. In: Proceedings of SIGIR 2008, pp. 675–682. ACM, New York (2008)
Teevan, J., Dumais, S.T., Horvitz, E.: Potential for personalization. ACM Transactions on Computer-Human Interaction (TOCHI) 17, 1–31 (2010)
Keim, D., Andrienko, G., Fekete, J.D., Görg, C., Kohlhammer, J., Melançon, G.: Information visualization, pp. 154–175. Springer, Heidelberg (2008)
Card, S.K., Mackinlay, J.: The structure of the information visualization design space. In: Proceedings of InfoVis 1997, pp. 92–99. IEEE Computer Society, Washington, DC, USA (1997)
Shneiderman, B.: The eyes have it: A task by data type taxonomy for information visualizations. In: Proceedings of the 1996 IEEE Symposium on Visual Languages, pp. 336–343. IEEE Computer Society, Washington, DC, USA (1996)
Keim, D., Kohlhammer, J., Santucci, G., Mansmann, F., Wanner, F., Schäfer, M.: Visual Analytics Challenges. In: Proceedings of the eChallenges 2009 (2009)
Seo, J., Shneiderman, B.: A rank-by-feature framework for interactive exploration of multidimensional data. Information Visualization 4, 96–113 (2005)
Derthick, M., Christel, M.G., Hauptmann, A.G., Wactlar, H.D.: Constant density displays using diversity sampling. In: Proceedings of InfoVis 2003, pp. 137–144. IEEE Computer Society, Washington, DC, USA (2003)
Banks, D., Over, P., Zhang, N.-F.: Blind Men and Elephants: Six Approaches to TREC data. Information Retrieval 1, 7–34 (1999)
Sormunen, E., Hokkanen, S., Kangaslampi, P., Pyy, P., Sepponen, B.: Query performance analyser -: a web-based tool for ir research and instruction. In: Proceedings of SIGIR 2002, p. 450. ACM, New York (2002)
Ferro, N., Sabetta, A., Santucci, G., Tino, G., Veltri, F.: Visual comparison of Ranked Result Cumulated Gains. In: Proceedings of EuroVA 2011 (2011)
Di Buccio, E., Dussin, M., Ferro, N., Masiero, I., Santucci, G., Tino, G.: Interactive analysis and exploration of experimental evaluation results. In: Proceedings of EuroHCIR 2011 (to appear, 2011)
Melucci, M.: Weighted rank correlation in information retrieval evaluation. In: Lee, G.G., Song, D., Lin, C.-Y., Aizawa, A., Kuriyama, K., Yoshioka, M., Sakai, T. (eds.) AIRS 2009. LNCS, vol. 5839, pp. 75–86. Springer, Heidelberg (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Di Buccio, E., Dussin, M., Ferro, N., Masiero, I., Santucci, G., Tino, G. (2011). To Re-rank or to Re-query: Can Visual Analytics Solve This Dilemma?. In: Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., de Rijke, M. (eds) Multilingual and Multimodal Information Access Evaluation. CLEF 2011. Lecture Notes in Computer Science, vol 6941. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23708-9_14
Download citation
DOI: https://doi.org/10.1007/978-3-642-23708-9_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-23707-2
Online ISBN: 978-3-642-23708-9
eBook Packages: Computer ScienceComputer Science (R0)