Justin Zobel
Alistair Moffat
Abstract
The concept of recall has been one of the key elements of system measurement throughout the history of information retrieval, despite the fact that there are many unanswered questions as to its value. In this essay, we review those questions and explore several further issues that affect the usefulness of recall. In particular, we ask whether it is reasonable to expect to be able to measure recall; whether some researchers are conflating the concepts of recall and answer set cardinality; and whether it is plausible that a user would rely on a belief that a system is "high recall" to deeply explore an answer list. Combined with earlier observations about the unknowability of recall, and the lack of a plausible user model in which recall is a measure of satisfaction, we conclude that use of recall as a measure of the effectiveness of ranked querying is indefensible.
- W.S. Cooper. Expected search length: A single measure of retrieval based on the weak ordering action of retrieval systems. American Documentation, 19(1):30--41, January 1968.Google Scholar
Cross Ref
- W.S. Cooper. On selecting a measure of retrieval effectiveness: Part I, the 'subjective' philosophy of evaluation. Jour. of the American Society for Information Science, 24:87--100, March 1973.Google ScholarCross Ref
- A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems, 27(1):(2)1--27, December 2008. Google ScholarDigital Library
- T. Saracevic. Evaluation of evaluation in information retrieval. In E.A. Fox, P. Ingwersen, and R. Fidel, editors, Proc. Eighteenth Annual International ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 138--146, Seattle, Washington, July 1995. ACM Press, New York. Google ScholarDigital Library
- J. Zobel. How reliable are the results of large-scale information retrieval experiments? In W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, and J. Zobel, editors, Proc. Twenty-First Annual International ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 307--314, Melbourne, Australia, August 1998. ACM Press, New York. Google ScholarDigital Library
Index Terms
- Against recall: is it persistence, cardinality, density, coverage, or totality?
Recommendations
Approximating maxmin strategies in imperfect recall games using A-loss recall property
AbstractExtensive-form games with imperfect recall are an important model of dynamic games where the players are allowed to forget previously known information. Often, imperfect recall games result from an abstraction algorithm that simplifies ...
Solving Imperfect Recall Games
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent SystemsImperfect recall games remain an unexplored part of the game theory, even though recent results in abstraction algorithms show that the imperfect recall might be the key to solving the immense games found in the real world efficiently. Our research ...
Interaction (f)or Recall? Evaluating Recall on Large Screens
MobileHCI '20: 22nd International Conference on Human-Computer Interaction with Mobile Devices and ServicesRemembering content (recall) is relevant for information and educational applications as well as advertisement. Previous research showed that the recall on large screens is increased when interacting with body gestures. Based on this finding we ...
Comments