research-article

Visual interactive failure analysis: supporting users in information retrieval evaluation

Authors:

Marco Angelini,

Gianmaria Silvello,

Giuseppe SantucciAuthors Info & Claims

IIIX '12: Proceedings of the 4th Information Interaction in Context Symposium

Pages 194 - 203

https://doi.org/10.1145/2362724.2362757

Published: 21 August 2012 Publication History

Abstract

Measuring is a key to scientific progress. This is particularly true for research concerning complex systems, whether natural or human-built. Multilingual and multimedia information access systems, such as search engines, are increasingly complex: they need to satisfy diverse user needs and support challenging tasks. Their development calls for proper evaluation methodologies to ensure that they meet the expected user requirements and provide the desired effectiveness. In this context, failure analysis is crucial to understand the behaviour of complex systems. Unfortunately, this is an especially challenging activity, requiring vast amounts of human effort to inspect query-by-query the output of a system in order to understand what went well or bad. It is therefore fundamental to provide automated tools to examine system behaviour, both visually and analytically. Moreover, once you understand the reason behind a failure, you still need to conduct a "what-if" analysis to understand what among the different possible solutions is most promising and effective before actually starting to modify your system. This paper provides an analytical model for examining performances of IR systems, based on the discounted cumulative gain family of metrics, and visualization for interacting and exploring the performances of the system under examination. Moreover, we propose machine learning approach to learn the ranking model of the examined system in order to be able to conduct a "what-if" analysis and visually explore what can happen if you adopt a given solution before having to actually implement it.

References

[1]

{Banks et al., 1999} Banks, D., Over, P., and Zhang, N.-F. (1999). Blind Men and Elephants: Six Approaches to TREC data. Information Retrieval, 1: 7--34.

Digital Library

[2]

{Berkhin, 2006} Berkhin, P. (2006). A Survey of Clustering Data Mining Techniques. In Kogan, J., Nicholas, C., and Teboulle, M., editors, Grouping Multidimensional Data, pages 25--71. Springer-Verlag, Heidelberg, Germany.

[3]

{Derthick et al., 2003a} Derthick, M., Christel, M. G., Hauptmann, A. G., and Wactlar, H. D. (2003a). Constant density displays using diversity sampling. In Proceedings of InfoVis'03, pages 137--144, Washington, DC, USA. IEEE Computer Society.

Digital Library

[4]

{Derthick et al., 2003b} Derthick, M., Christel, M. G., Hauptmann, A. G., and Wactlar, H. D. (2003b). Constant density displays using diversity sampling. In Proceedings of the IEEE Information Visualization, pages 137--144.

Digital Library

[5]

{Freund et al., 2003} Freund, Y., Iyer, R., Schapire, R. E., and Singer, Y. (2003). An Efficient Boosting Algorithm for Combining Preferences. Journal of Machine Learning Research, 4(Nov): 933--969.

Digital Library

[6]

{Geng et al., 2007} Geng, X., Liu, T.-Y., Qin, T., and Li, H. (2007). Feature Selection for Ranking. In Kraaij, W., de Vries, A. P., Clarke, C. L. A., Fuhr, N., and Kando, N., editors, Proc. 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007), pages 407--414. ACM Press, New York, USA.

Digital Library

[7]

{Harman and Buckley, 2009} Harman, D. and Buckley, C. (2009). Overview of the reliable information access workshop. Information Retrieval, 12(6): 615--641.

Digital Library

[8]

{Järvelin and Kekäläinen, 2002} Järvelin, K. and Kekäläinen, J. (2002). Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information System (TOIS), 20(4): 422--446.

Digital Library

[9]

{Keskustalo et al., 2008} Keskustalo, H., Järvelin, K., Pirkola, A., and Kekäläinen, J. (2008). Intuition-Supporting Visualization of User's Performance Based on Explicit Negative Higher-Order Relevance. In Chua, T.-S., Leong, M.-K., Oard, D. W., and Sebastiani, F., editors, Proc. 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2008), pages 675--682. ACM Press, New York, USA.

Digital Library

[10]

{Korfhage, 1997} Korfhage, R. R. (1997). Information Storage and Retrieval. Wiley Computer Publishing, John Wiley & Sons, Inc., USA.

Digital Library

[11]

{Liu, 2009} Liu, T.-Y. (2009). Learning to Rank for Information Retrieval. Foundations and Trends in Information Retrieval, 3(3): 225--331.

Digital Library

[12]

{Liu et al., 2007} Liu, T.-Y. Y., Xu, J., Qin, T., Xiong, W., and Li, H. (2007). LETOR: Benchmark Dataset for Research on Learning to Rank for Information Retrieval. In Joachims, T., Li, H., Liu, T.-Y., and Zhai, C., editors, SIGIR 2007 Workshop on Learning to Rank for Information Retrieval.

[13]

{Seo and Shneiderman, 2004} Seo, J. and Shneiderman, B. (2004). A rank-by-feature framework for interactive exploration of multidimensional data. In Proceedings of the IEEE Information Visualization, pages 65--72.

Digital Library

[14]

{Seo and Shneiderman, 2005} Seo, J. and Shneiderman, B. (2005). A rank-by-feature framework for interactive exploration of multidimensional data. Information Visualization, 4: 96--113.

Digital Library

[15]

{Sormunen et al., 2002} Sormunen, E., Hokkanen, S., Kangaslampi, P., Pyy, P., and Sepponen, B. (2002). Query Performance Analyser -- a Web-based tool for IR research and instruction. In Järvelin, K., Beaulieu, M., Baeza-Yates, R., and Hyon Myaeng, S., editors, Proc. 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002), page 450. ACM Press, New York, USA.

Digital Library

[16]

{Teevan et al., 2010} Teevan, J., Dumais, S. T., and Horvitz, E. (2010). Potential for personalization. ACM Transactions on Computer-Human Interaction (TOCHI), 17(1): 1--31.

Digital Library

[17]

{van Rijsbergen, 1979} van Rijsbergen, C. J. (1979). Information Retrieval. Butterworths, London, England, 2nd edition.

Digital Library

[18]

{Voorhees and Harman, 1999} Voorhees, E. and Harman, D. (1999). Overview of the Seventh Text REtrieval Conference (TREC-7). In NIST Special Publication 500--242: The Seventh Text REtrieval Conference (TREC 7). Springer-Verlag, Heidelberg, Germany.

Cited By

Jarvelin KSormunen E(2024)A Blueprint of IR Evaluation Integrating Task and User CharacteristicsACM Transactions on Information Systems10.1145/367516242:6(1-38)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3675162
Ioannakis GKoutsoudis APratikakis IChamzas C(2018)RETRIEVAL—An Online Performance Evaluation Tool for Information Retrieval MethodsIEEE Transactions on Multimedia10.1109/TMM.2017.271619320:1(119-127)Online publication date: 25-Dec-2018
https://dl.acm.org/doi/10.1109/TMM.2017.2716193
Angelini MFerro NSantucci GSilvello G(2018)VIRTUEJournal of Visual Languages and Computing10.1016/j.jvlc.2013.12.00325:4(394-413)Online publication date: 27-Dec-2018
https://dl.acm.org/doi/10.1016/j.jvlc.2013.12.003
Show More Cited By

Index Terms

Visual interactive failure analysis: supporting users in information retrieval evaluation

Recommendations

DESIRE 2011: first international workshop on data infrastructures for supporting information retrieval evaluation
CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge management

The workshop focuses on the three areas of interest to CIKM to discuss how to envisage and design evaluation infrastructures able to store, manage, and make accessible the scientific data and knowledge of interest for advancing the evaluation of ...
VIRTUE: A visual tool for information retrieval performance evaluation and failure analysis

Objective: Information Retrieval (IR) is strongly rooted in experimentation where new and better ways to measure and interpret the behavior of a system are key to scientific advancement. This paper presents an innovative visualization environment: ...
VizDeck: a card game metaphor for fast visual data exploration
CHI EA '12: CHI '12 Extended Abstracts on Human Factors in Computing Systems

Scientists in all fields are acquiring data at a rate that is challenging the limits of human cognitive capacity. At the same time, researchers' attention is increasingly claimed by ever more diverse demands on their time. Visual perception is the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

IIIX '12: Proceedings of the 4th Information Interaction in Context Symposium

August 2012

347 pages

ISBN:9781450312820

DOI:10.1145/2362724

General Chairs:
Jaap Kamps,
Wessel Kraaij,
Program Chair:
Norbert Fuhr
University of Duisburg-Essen

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

University of Amsterdam: The University of Amsterdam

In-Cooperation

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 August 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IIiX'12

Sponsor:

University of Amsterdam

IIiX'12: Information Interaction in Context: 2012

August 21 - 24, 2012

Nijmegen, The Netherlands

Acceptance Rates

Overall Acceptance Rate 21 of 45 submissions, 47%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
131
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Jarvelin KSormunen E(2024)A Blueprint of IR Evaluation Integrating Task and User CharacteristicsACM Transactions on Information Systems10.1145/367516242:6(1-38)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3675162
Ioannakis GKoutsoudis APratikakis IChamzas C(2018)RETRIEVAL—An Online Performance Evaluation Tool for Information Retrieval MethodsIEEE Transactions on Multimedia10.1109/TMM.2017.271619320:1(119-127)Online publication date: 25-Dec-2018
https://dl.acm.org/doi/10.1109/TMM.2017.2716193
Angelini MFerro NSantucci GSilvello G(2018)VIRTUEJournal of Visual Languages and Computing10.1016/j.jvlc.2013.12.00325:4(394-413)Online publication date: 27-Dec-2018
https://dl.acm.org/doi/10.1016/j.jvlc.2013.12.003
Ferro NSilvello GKeskustalo HPirkola AJärvelin K(2016)The twist measure for IR evaluationJournal of the Association for Information Science and Technology10.1002/asi.2341667:3(620-648)Online publication date: 1-Mar-2016
https://dl.acm.org/doi/10.1002/asi.23416
Angelini MFerro NSantucci GSilvello G(2014)A Visual Interactive Environment for Making Sense of Experimental DataProceedings of the 36th European Conference on IR Research on Advances in Information Retrieval - Volume 841610.5555/2964060.2964135(767-770)Online publication date: 13-Apr-2014
https://dl.acm.org/doi/10.5555/2964060.2964135
Angelini MFerro NSantucci GSilvello G(2013)Improving Ranking Evaluation Employing Visual AnalyticsProceedings of the 4th International Conference on Information Access Evaluation. Multilinguality, Multimodality, and Visualization - Volume 813810.1007/978-3-642-40802-1_4(29-40)Online publication date: 23-Sep-2013
https://dl.acm.org/doi/10.1007/978-3-642-40802-1_4
Agosti MBerendsen RBogers TBraschler MBuitelaar PChoukri KMaria Di Nunzio GFerro NForner PHanbury AHeppin KHansen PJärvelin ALarsen BLupu MMasiero IMüller HPeruzzo SPetras VPiroi Fde Rijke MSantucci GSilvello GToms E(2012)PROMISE retreat report prospects and opportunities for information access evaluationACM SIGIR Forum10.1145/2422256.242226546:2(60-84)Online publication date: 21-Dec-2012
https://dl.acm.org/doi/10.1145/2422256.2422265

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten