Abstract
We present a user-centred, task-oriented, comparative evaluation of two query-based document skimming tools. ProfileSkim bases within-document retrieval on computing a relevance profile for a document and query; FindSkim provides similar functionality to the web browser Find-command. A novel simulated work task was devised, where experiment participants are asked to identify (index) relevant pages of an electronic book, given subjects from the existing book index. This subject index provides the ground truth, against which the indexing results can be compared. Our major hypothesis was confirmed, namely ProfileSkim proved significantly more efficient than FindSkim, as measured by time for task. Moreover, indexing task effectiveness, measured by typical IR measures, demonstrated that ProfileSkim was better than FindSkim in identifying relevant pages, although not significantly so. The experiments confirm the potential of relevance profiling to improve query-based document skimming, which should prove highly beneficial for users trying to identify relevant information within long documents.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Kaszkiel, M., Zobel, J.: Passage Retrieval Revisited. In: Proceedings of the Twentieth International ACM-SIGIR Conference on Research and Development in Information Retrieval. Philadelphia. ACM Press (1997) 178–185
Hearst, M.A.: TileBars: Visualization of Term Distribution Information in Full Text Information Access. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), Denver, CO, (1995)
Harper, D.J., Coulthard, S., Sun, Y.: A Language Modelling Approach to Relevance Profiling for Document Browsing. In: Proceedings of the Joint Conference on Digital Libraries. Oregon, USA (2002) 76–83
Borlund, P., Ingwersen, P.: The Development of a Method for the Evaluation of Interactive Information Retrieval Systems. Journal of Documentation. 53(3) (1997) 225–250
Beaulieu, M., Robertson, S.E. and Rasmussen, E.: Evaluating interactive systems in TREC. Journal of the American Society for Information Science. 47(1) (1996) 85–94
Hersh, W., Pentecost, J., Hickam, D.: A Task-Oriented Approach to Information Retrieval Evaluation. Journal of the American Society for Information Science 47(1) (1996) 50–56
Jose, J., Furner, J., Harper, D.J.: Spatial Querying for Image Retrieval: A User-Oriented Evaluation. In: Proceedings of the Twenty First International ACM-SIGIR Conference on Research and Development in Information Retrieval, ACM Press (1998) 232–240
Dunlop, M. (ed):. Proceedings of the Second Mira Workshop, Technical Report TR-1997-2. Department of Computing Science, University of Glasgow, Glasgow (1996). Available online at URL: http://wu.dcs.gla.ac.uk/mira/workshops/padua_procs/ .
Ponte, J., Croft, W.B.: A Language Modeling Approach to Information Retrieval. In: Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press (1998) 275–281
Song, F., Croft, W.B.: A General Language Model for Information Retrieval. In: Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press (1999) 279–280
Green, T.R.G.: Describing Information Artifacts with Cognitive Dimensions and Structure Maps. In: Diaper, D., Hammond, N.V. (eds.): Proceedings of the HCI’91 Conference on People and Computers VI. Cambridge University Press, Cambridge (1991)
Hersh, W.R., Over, P.: TREC 2001 Interactive Track Structure, Proceedings of the Text Retrieval Conference (TREC) 2001, Gaithersburg, MD, (2001) 38–41
Harman, D.: Overview of the First Text REtrieval Conference (TREC-1), National Institute of Standards and Technology, Gaithersburg, Maryland (1992) 309–318
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Harper, D.J., Koychev, I., Sun, Y. (2003). Query-Based Document Skimming: A User-Centred Evaluation of Relevance Profiling. In: Sebastiani, F. (eds) Advances in Information Retrieval. ECIR 2003. Lecture Notes in Computer Science, vol 2633. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-36618-0_27
Download citation
DOI: https://doi.org/10.1007/3-540-36618-0_27
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-01274-0
Online ISBN: 978-3-540-36618-8
eBook Packages: Springer Book Archive