Abstract
The ultimate goal of information retrieval (IR) research is to create ways to support humans to better access information in order to better carry out their (work) tasks. Because of this, IR research has a primarily technological interest in knowledge creation – how to find information (better)? IR research therefore has a constructive aspect (to create novel systems) and an evaluative aspect (are they any good?). Evaluation is sometimes referred to as a hallmark and distinctive feature of IR research. No claim on IR system performance is granted any merit unless proven through evaluation. Technological innovation alone is not sufficient. In fact, much research in IR deals with IR evaluation and its methodology.
Evaluation, in general, is the systematic determination of merit and significance of something using criteria against some standards. Evaluation therefore requires some object that is evaluated and some goal that should be achieved or served. In IR, both can be set in many ways. The object usually is an IR system – but what is an IR system? The goal is typically the quality of the retrieved result – but what is the retrieved result and how does one measure quality? These questions can be answered in alternative ways leading to different kinds of IR evaluation.
Practical life with all its variability is difficult and expensive to investigate. Therefore surrogate and more easily measurable goals are employed in IR evaluation, typically the quality of the ranked result list instead of the work task result. The task performance process may also be cut down from a work task to a search task and down to running an individual query in a test collection. This simplification has led to standardization of research designs and tremendous success in IR research. However, as the goals and systems drift farther away from the practical life condition, one needs to ask, whether the findings still best serve the initial goal of evaluation (supporting human performance)? If means (outputs) replace ends (outcomes), one runs the risk of sub-optimization.
It is important to evaluate all subsystems of information retrieval processes, in addition to the search engines. Through a wider perspective one may be able to put the subsystems and their contributions in relation with each other. We will discuss nested IR evaluation frameworks ranging from IR system centered evaluation to work-task based evaluation. We will also point to the Pandora’s box of problems that the enlargement of the scope of research entails. Is science at risk here?
The contributions of a research area, in addition to constructive and evaluative contributions, may be generally empirical, theoretical and methodological. Why should anyone in IR care about anything beyond IR experimentation (i.e. evaluation) using test collections? The Cranfield model seeks to relate texts (documents), queries, their representations and matching to topical relevance in ranked output. Who relates this, and a range of possible other contributing factors, to outcomes in search task performance or work task performance? The talk will outline some possibilities for descriptive, explanatory and theoretical research in IR. As an example of descriptive research, we will discuss information access in task processes. Regarding explanatory and theoretical research, we look at unit theories that connect work task stages and properties to information need properties, information sources, and searching. Such studies do not solve a technical problem, nor evaluate any particular technique, and may therefore be considered unpractical. However, they may identify mechanisms that mediate between IR processes and task outcomes and position factors in the processes of information access into a balanced perspective. Therefore they may help focus research efforts on technical problems or evaluation.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Byström, K., Järvelin, K.: Task Complexity Affects Information Seeking and Use. Information Processing & Management 31(2), 191–213 (1995)
Ingwersen, P., Järvelin, K.: The Turn: Integration of Information Seeking and Retrieval in Context. Springer, Dortrecht (2005)
Järvelin, K.: An Analysis of Two Approaches in Information Retrieval: From Frameworks to Study Designs. Journal of the American Society for Information Science and Technology (JASIST) 58(7), 971–986 (2007)
Järvelin, K.: Explaining User Performance in Information Retrieval: Challenges to IR evaluation. In: Azzopardi, L., Kazai, G., Robertson, S., Rüger, S., Shokouhi, M., Song, D., Yilmaz, E. (eds.) ICTIR 2009. LNCS, vol. 5766, pp. 289–296. Springer, Heidelberg (2009)
Kumpulainen, S., Järvelin, K.: Information Interaction in Molecular Medicine: Integrated Use of Multiple Channels. In: Belkin, N., et al. (eds.) Proceedings of the Information Interaction in Context Conference (IIiX 2010), pp. 95–104. ACM, New York (2010)
Vakkari, P.: A theory of the task-based information retrieval process: A summary and generalization of a longitudinal study. Journal of Documentation 57(1), 44–60 (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Järvelin, K. (2011). IR Research: Systems, Interaction, Evaluation and Theories. In: Clough, P., et al. Advances in Information Retrieval. ECIR 2011. Lecture Notes in Computer Science, vol 6611. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-20161-5_1
Download citation
DOI: https://doi.org/10.1007/978-3-642-20161-5_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-20160-8
Online ISBN: 978-3-642-20161-5
eBook Packages: Computer ScienceComputer Science (R0)