skip to main content
10.1145/1835449.1835551acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article

PRES: a score metric for evaluating recall-oriented information retrieval applications

Published: 19 July 2010 Publication History

Abstract

Information retrieval (IR) evaluation scores are generally designed to measure the effectiveness with which relevant documents are identified and retrieved. Many scores have been proposed for this purpose over the years. These have primarily focused on aspects of precision and recall, and while these are often discussed with equal importance, in practice most attention has been given to precision focused metrics. Even for recall-oriented IR tasks of growing importance, such as patent retrieval, these precision based scores remain the primary evaluation measures. Our study examines different evaluation measures for a recall-oriented patent retrieval task and demonstrates the limitations of the current scores in comparing different IR systems for this task. We introduce PRES, a novel evaluation metric for this type of application taking account of recall and the user's search effort. The behaviour of PRES is demonstrated on 48 runs from the CLEF-IP 2009 patent retrieval track. A full analysis of the performance of PRES shows its suitability for measuring the retrieval effectiveness of systems from a recall focused perspective taking into account the user's expected search effort.

References

[1]
Ali M. S., Consens, M. P., Kazai, G., and Lalmas, M. Structural relevance: A common basis for the evaluation of structured document retrieval. In Proceedings of CIKM '08, pages 1153--1162, 2008.
[2]
Aslam J. A., and E. Yilmaz. Estimating average precision with incomplete and imperfect judgments. In Proceedings of CIKM' 06, page 102--111, 2006.
[3]
Azzopardi L., de Rijke, M., and K. Balog. Building simulated queries for known-item topics: an analysis using six european languages. In Proceedings of SIGIR '07, pages 455--462, 2007.
[4]
Azzopardi, L. and Vinay, V. Retrievability. An evaluation measure for higher order information access tasks. In Proccedings of CIKM '08, pages 1425--1426, 2008.
[5]
Baeza-Yates, J., and Ribeiro-Neto, B. Modern Information Retrieval. Addison Wesley, 1999.
[6]
Bashir, S., and Rauber A. Analyzing Document Retrievability in Patent Retrieval Settings. In Proceedings of Database and Expert Systems Applications (DEXA 2009), pages 753--760, 2009.
[7]
Buckley, C., and Voorhees, E. M. Evaluating Evaluation Measure Stability. In Proceedings of SIGIR 2000, pages 33--40, 2000.
[8]
Buckley, C., Dimmick, D., Soboroff, I., and E. Voorhees. Bias and the limits of pooling. In Proceedings of SIGIR '06, pages 619--620, 2006.
[9]
Buckley, C., and Voorhees, E. M. Retrieval evaluation with incomplete information. In Proceedings of SIGIR '04, pages 25--32, 2004.
[10]
Carterette, B., Bennett, P. N. Chickering, D. M., and Dumais, S. T. Here or There: Preference Judgments for Relevance. In Proceedings of ECIR '08, pages 16--27, 2008.
[11]
Cleverdon, C. The Cranfield Tests on Index Language Devices. In: Sparck Jones, K. and Willett, P. (eds.). Readings in Information Retrieval, pages 47--59, Morgan Kaufmann, 1997.
[12]
Hull, D. Using statistical testing in the evaluation of retrieval experiments. In Proceedings of SIGIR '93, pages 329--338, 1993.
[13]
Fujii, A., Iwayama, M., and Kando, N. Overview of Patent Retrieval Task at NTCIR-4. In Proceedings of the 4th NTCIR Workshop, 2004.
[14]
Graf, E., and Azzopardi, L. A methodology for building a patent test collection for prior art search. In Proceedings of the 2nd EVIA Workshop, pages 60--71, 2008.
[15]
Jordan, C., Watters, C., and Gao, Q. Using controlled query generation to evaluate blind relevance feedback algorithms. In Proceedings of JCDL '06, pages 286--295, 2006.
[16]
Kamps, J., Pehcevski, J., Kazai, G., Lalmas, M., and Robertson, S. INEX 2007 evaluation measures. In Proceedings of INEX '07, pages 24--33, 2007.
[17]
Kendall, M. A new measure of rank correlation. Biometrika, 30(1/2):81--93, 1938.
[18]
Mandl, T. Recent developments in the evaluation of information retrieval systems: moving toward diversity and practical applications. Informatica, 32:27--38, 2008.
[19]
Moffat, A., and Zobel, J. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst. 27(1):1--27, 2008.
[20]
Oard, D. W., Hedin, B., Tomlinson, S., and Baron, J. R. Overview of the TREC 2008 legal track. In Proceedings of TREC 2008, 2008.
[21]
van Rijsbergen, C. J. Information Retrieval, 2nd edition. Butterworths, 1979.
[22]
Robertson S. E. The parametric description of the retrieval tests. Part 2: Overall measures. Journal of Documentation, 25(2):93--107, 1969.
[23]
Robertson, S. A new interpretation of average precision. In Proceedings of SIGIR '08, pages 689--690, 2008.
[24]
Rocchio J. Performance indices for document retrieval systems. In Information storage and retrieval, Computation Laboratory of Harvard University, Cambridge, MA, 1964.
[25]
Roda G., Tait J., Piroi F., and Zenz V. CLEF-IP 2009: retrieval experiments in the Intellectual Property domain. In Proceedings of CLEF '09, 2009.
[26]
Tague J., Nelson, M., and Wu, H. Problems in the simulation of bibliographic retrieval systems. In Proceeding of SIGIR '81, pages 66--71, 1981.
[27]
Tomlinson S., Oard, D. W., Baron, J. R., and Thompson, P. Overview of the TREC 2007 Legal Track. In Proceedings of TREC 2007, 2007.
[28]
Voorhees, E. M., and Tice, D. M. The TREC-8 Question Answering Track Evaluation. In Proceedings of TREC 1999, pages 77--82, 1999.
[29]
Voorhees, E. M. The Philosophy of Information Retrieval Evaluation. In Evaluation of Cross-Language Information Retrieval System, Proceedings of CLEF '02, pages 355--370, 2002.
[30]
Voorhees, E. M. The TREC robust retrieval track. In SIGIR Forum 39(1):11--20, 2005.
[31]
Xue, X., and Croft W. B. Automatic Query Generation for Patent Search. In Proceedings of CIKM'09, pages 2037--2040, 2009.
[32]
Zhu, J., and Tait, J. A proposal for chemical information retrieval evaluation. In In Proceedings of the 1st ACM Workshop on Patent Information Retrieval at CIKM '08, pages 15--18, 2008.

Cited By

View all
  • (2024)Research on Keyword-Based Element Extraction for Chinese Patent Retrieval2024 International Conference on Asian Language Processing (IALP)10.1109/IALP63756.2024.10661116(67-73)Online publication date: 4-Aug-2024
  • (2023)Outcome-based Evaluation of Systematic Review AutomationProceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3578337.3605135(125-133)Online publication date: 9-Aug-2023
  • (2023)Data Avatars: A Theory-Guided Design and Assessment for Multidimensional Data VisualizationInformation & Management10.1016/j.im.2023.103911(103911)Online publication date: Dec-2023
  • Show More Cited By

Index Terms

  1. PRES: a score metric for evaluating recall-oriented information retrieval applications

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
    July 2010
    944 pages
    ISBN:9781450301534
    DOI:10.1145/1835449
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 19 July 2010

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. PRES
    2. evaluation metric
    3. patent retrieval
    4. recall-oriented information retrieval

    Qualifiers

    • Research-article

    Conference

    SIGIR '10
    Sponsor:

    Acceptance Rates

    SIGIR '10 Paper Acceptance Rate 87 of 520 submissions, 17%;
    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)52
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 20 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Research on Keyword-Based Element Extraction for Chinese Patent Retrieval2024 International Conference on Asian Language Processing (IALP)10.1109/IALP63756.2024.10661116(67-73)Online publication date: 4-Aug-2024
    • (2023)Outcome-based Evaluation of Systematic Review AutomationProceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3578337.3605135(125-133)Online publication date: 9-Aug-2023
    • (2023)Data Avatars: A Theory-Guided Design and Assessment for Multidimensional Data VisualizationInformation & Management10.1016/j.im.2023.103911(103911)Online publication date: Dec-2023
    • (2022)Rev: A Video Engine for Object Re-identification at the City Scale2022 IEEE/ACM 7th Symposium on Edge Computing (SEC)10.1109/SEC54971.2022.00022(189-202)Online publication date: Dec-2022
    • (2022)Dynamic Information Retrieval ModelingundefinedOnline publication date: 10-Mar-2022
    • (2021)Aplicación de inteligencia artificial para monitorear el uso de mascarillas de protecciónRevista Científica General José María Córdova10.21830/19006586.72519:33(205-222)Online publication date: 1-Jan-2021
    • (2021)AdaSimProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482316(1528-1537)Online publication date: 26-Oct-2021
    • (2021)Results Merging in the Patent Domain24th Pan-Hellenic Conference on Informatics10.1145/3437120.3437313(229-232)Online publication date: 4-Mar-2021
    • (2021)JacSim*: An Effective and Efficient Solution to the Pairwise Normalization Problem in SimRankIEEE Access10.1109/ACCESS.2021.31231149(146038-146049)Online publication date: 2021
    • (2020)On Investigating Both Effectiveness and Efficiency of Embedding Methods in Task of Similarity Computation of Nodes in GraphsApplied Sciences10.3390/app1101016211:1(162)Online publication date: 26-Dec-2020
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media