Abstract
This paper presents a holistic evaluation of an operational information system that employs the Boolean search technique. An equal focus is laid on both the system (system perspective) and its users (user perspective) in the actual environment where the system and its users are functioning (contextuality). In addition to these research objectives, the study has a methodological objective to test an evaluation approach developed by Borlund [1] in a real life setting. Our evaluation methodology involves triangulation (pre-search questionnaires; search log; post-interviewing) as well as novel interactive performance measures, such as the Ranked Half-Life measure and the Satisfaction and Novelty perception by users supplementing the traditional Precision. The study confirms the finding of earlier research and reveals the discrepancy between the evaluation results according to the system and the user perspectives. More specifically, the system performed better when evaluated from the user perspective than from the system perspective.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Borlund, P.: Evaluation of interactive information retrieval systems. Åbo Akademi University Press, Åbo (2000)
Cooper, W.S.: A definition of relevance for information retrieval. Information Storage and Retrieval 7(1), 19–37 (1971)
Saracevic, T.: Relevance reconsidered 1996. In: Ingwersen, P., Pors, N.O. (eds.) Information Science: Integration in Perspective (CoLIS2), pp. 201–218. Royal School of Librarianship, Copenhagen (1996)
Wilson, P.: Situational relevance. Information Storage and Retrieval 9, 457–469 (1973)
Ingwersen, P.: Cognitive perspectives of information retrieval interaction: elements of a cognitive IR theory. Journal of Documentation 52(1), 3–50 (1996)
Ingwersen, P.: Users in Context. In: Agosti, Crestani, Pasi (eds.) Lectures on Information Retrieval, pp. 157–178. Springer, Bonn (2001)
Taylor, R.S.: Information use environments. In: Dervin, B., Voigt, M. (eds.) Progress in communication sciences, pp. 217–255. Ablex, Norwood (1991)
Byström, K., Hansen, P.: Work tasks as units for analysis in information seeking and retrieval studies. In: Bruce, H., Fidel, R., Ingwersen, P., Vakkari, P. (eds.) Emerging Frameworks and Methods, Libraries Unlimited, Greenwood Village, CO, pp. 239–251 (2002)
Mizzaro, S.: How many relevances in information retrieval? Interacting with Computers 10, 303–320 (1998)
Reid, J.: A task-oriented non-interactive evaluation methodology for information retrieval systems. Information Retrieval 2(1), 113–127 (2000)
Schamber, L.: Relevance and information behavior. In: Williams, Martha 0.E. (eds.) Annual Review of Information Science and Technology (ARIST), vol. 29, pp. 3–48 (1994)
Byström, K.: Information and information sources in tasks of varying complexity. Journal of American Society of Information Science and Technology 53(7), 581–591 (2002)
Webster, J., Trevino, L.K.: Rational and social theories as complementary explanations of communication media choices: Two policy-capturing studies. Academy of Management Journal 38(6), 1544–1572 (1995)
Harter, S., Hert, C.: Evaluation of Information Retrieval Systems: Approaches, Issues, and Method. In: Williams, I.i., Martha, E. (eds.) Annual Review of Information Science and Technology (ARIST), vol. 32, pp. 3–94 (1997)
Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval. Addison-Wesley, Reading (1999)
Stephen, P., Hornby, S.: Simple statistics: for library and information professionals, 2nd edn. Library Association Publishing, London (1997)
Kekäläinen, Järvelin: Evaluating Information Retrieval Systems under the Challenges of Interaction and Multi-Dimensional Dynamic Relevance. In: Ingåri Bruce, Harry, Fidel, Raya, Ingwersen, Peter, Vakkari, Pertti (eds.) Proceedings of the 4 th CoLIS Conference. Greenwood Village, CO: Libraries Unlimited. s. pp. 253-270 (2002)
Robertson, Hancock-Beaulieu: On the Evaluation of IR Systems. Information Processing and Management 28(4), 457–466 (1992)
Schamber, L., Eisenberg, M., Nilan, M.: A re-examination of relevance: Toward a dynamic, situational definition. Information Processing & Management 26(6), 755–776 (1990)
Vakkari, P.: A Theory of the Task-based Information Retrieval Process. Journal of Documentation 57(1), 44–60 (2001)
Nicholas, D., Martin, H.: Should journalists search themselves (And what happens when they do?). In: Raitt, D.I., Jeapes, B. (eds.) Online information 1993: 17th International Online Information Meeting, London, Oxford, December 7-9, pp. 227–234. Learned Information (1993)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Blomgren, L., Vallo, H., Byström, K. (2004). Evaluation of an Information System in an Information Seeking Process. In: Heery, R., Lyon, L. (eds) Research and Advanced Technology for Digital Libraries. ECDL 2004. Lecture Notes in Computer Science, vol 3232. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30230-8_6
Download citation
DOI: https://doi.org/10.1007/978-3-540-30230-8_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23013-7
Online ISBN: 978-3-540-30230-8
eBook Packages: Springer Book Archive