Abstract
This study examines the use of an ontology as a search tool. Sixteen subjects created queries using Concept-based Information Retrieval Interface (CIRI) and a regular baseline IR interface. The simulated work task method was used to make the searching situations realistic. Subjects’ search experiences, queries and search results were examined. The numbers of search concepts and keys, as well as their overlap in the queries were investigated. The effectiveness of the CIRI and baseline queries was compared. An Ontology Index (OI) was calculated for all search tasks and the correlation between the OI and the overlap of search concepts and keys in queries was investigated. The number of search keys and concepts was higher in CIRI queries than in baseline interface queries. Also the overlap of search keys was higher among CIRI users than among baseline users. These both findings are due to CIRI’s expansion feature. There was no clear correlation between OI and overlap of search concepts and keys. The search results were evaluated with generalised precision and recall, and relevance scores based on individual relevance assessments. The baseline interface queries performed better in all comparisons, but the difference was statistically significant only in relevance scores based on individual relevance assessments.
Article PDF
Similar content being viewed by others
References
Allan, J., Callan, J., Croft, B., Ballesteros, L., Byrd, D., Swan, R., & Xu, J. (1997). INQUERY does battle with TREC-6. In: E. M., Voorhees, & D. K., Harman (Eds.), Proceedings of the Sixth Text REtrieval Conference (TREC-6) (pp. 169–206). Gaithersburg, MD, National Institute of Standards and Technology, special publication 500–240. http://trec.nist.gov/pubs/trec6/papers/umass-trec97.ps (visited October 26th, 2004).
Belkin, N. J., Cool, C., Kelly, D., Lin, S. J., Park, S. Y., Perez-Carballo, J., & Sikora, C. (2001). Iterative exploration, design and evaluation of support for query formulation in interactive information retrieval. Information Processing and Management, 37, 403–434.
Borlund, P. (2000). Evaluation of interactive information retrieval systems. Åbo: Åbo Akademi University Press.
Conover, W. J. (1999). Practical nonparametric statistics (3rd ed.). New York: John Wiley and Sons.
Efthimiadis, E. N. (1996). Query expansion. In: M. E., Williams (Ed.), Annual Review of Information Science and Technology. (vol, 31, pp. 121–187). Medford, NJ: Information Today.
Guarino, N. (1995). Formal ontology, conceptual analysis and knowledge representation. International Journal of Human and Computer Studies, 43, 625–640.
Iivonen, M. (1995). Hakulausekkeiden muotoilun yhdenmukaisuus onlineviitehaussa.Tampere: Acta Universitatis Tamperensis ser A, University of Tampere (Vol. 443).
Ingwersen, P. (2000). Users in Context. In: M., Agosti, F., Crestani, & G., Pasi (Eds.), ESSIR 2000. Lecture Notes in Computer Science. (vol. 1980, pp. 157–178). Berlin: Springer-Verlag.
ISO International Standard 2788. (1986). Documentation—Guidelines for the establishment and development of monolingual thesauri. International Organization for Standardization.
Joho, H., Sanderson, M., & Beaulieu, M. (2004). A study of user interaction with a concept-based interactive query expansion support tool. In: S., McDonald, & J., Tait (Eds.), Advances in Information Retrieval, Proceedings of 26th European Conference on IR Research, ECIR 2004. Lecture Notes in Computer Science. (vol. 2997, pp. 42–56). Berlin: Springer-Verlag.
Jones, S., Gatford, M., Robertson, S., Hancock-Beaulieu, M., & Secker, J. (1995). Interactive thesaurus navigation: Intelligence rules ok? Journal of the American Society for Information Science, 46, 52–59
Järvelin, K., Kekäläinen, J., & Niemi. T. (2001). ExpansionTool: Concept-based query expansion and construction. Information Retrieval, 4, 231–255.
Järvelin, K., Kristensen, J., Niemi, T., Sormunen, E., & Keskustalo, H. (1996). A deductive data model for query expansion. In: H.-P., Frei, D., Harman, P., Schäuble, & R. Wilkinson (Eds.), Proceedings of the 19th Annual International ACM—SIGIR Conference on Research and Development in Information Retrieval. (pp. 235–249). New York, NY: ACM Press.
Kekäläinen, J. (1999). The effects of query complexity, expansion and structure on retrieval performance in probabilistic text retrieval. Tampere: Acta Universitatis Tamperensis 678, University of Tampere.
Kekäläinen, J., & Järvelin, K. (2002). Using graded relevance assessments in IR evaluation. Journal of the American Society for Information Science and Technology, 53, 1120–1129.
Over, P. (2001). The TREC interactive track: an annotated bibliography. Information Processing & Management, 37, 369–381.
Saracevic, T. (1984). Measuring the degree of agreement between searchers. In: B., Flood, J., Witiak, & T. H., Hogan (Eds.), ASIS’84: Proceedings of the American Society for Information Science 47th annual meeting. (vol. 21, pp. 227–230). White Plains. NY: Knowledge Industry Publications.
Sihvonen, A., & Vakkari, P. (2004). Subject knowledge improves interactive query expansion assisted by a thesaurus. Journal of Documentation, 60, 673–690.
Sutcliffe, A. G., Ennis, M., & Watkinson, S. J. (2000). Empirical studies of end-user information searching. Journal of the American Society for Information Science, 51, 1211–1231.
Turtle, H. R. (1990). Inference networks for document retrieval (COINS Tech. Rep. No. 90–92). Computer and Information Science Department. University of Massachusetts.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Suomela, S., Kekäläinen, J. User evaluation of ontology as query construction tool. Inf Retrieval 9, 455–475 (2006). https://doi.org/10.1007/s10791-006-6387-3
Received:
Revised:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/s10791-006-6387-3