Skip to main content
SpringerLink
Log in

Consensus-based evaluation framework for distributed information retrieval systems

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Multi-agent systems have been attacking the challenges of information retrieval tasks on distributed environment. In this paper, we propose a consensus choice selection method based framework to evaluate the performance of cooperative information retrieval tasks of the multiple agents. Thereby, two well-known measurements, precision and recall, are extended to handle consensual closeness (i.e., local and global consensus) between the sets of retrieved results. We show that in a motivating example the proposed criteria are prone to solve the rigidity problem of classical precision and recall. More importantly, the retrieved results can be ranked with respect to the consensual score, and the ranking mechanism has been verified to be more reasonable.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Baeza-Yates R, Ribeiro-Neto B (1999) Modern information retrieval. Addison-Wesley, Reading

    Google Scholar 

  2. Borlund P (2003) The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Inform Res 8(3)

  3. Buckland M, Gey F (1994) The relationship between recall and precision. J Am Soc Inform Sci 45(1): 12–19

    Article  Google Scholar 

  4. Callan JP (2000) Distributed information retrieval, Chap. 5. Advances in Information Retrieval. Kluwer, Dordrecht, pp 127–150

  5. Callan JP et al (1995) Searching distributed collections with inference networks. In: Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR ’95), New York. ACM Press, New York, pp 21–28

  6. Chen H-H et al (2003) Overview of CLIR task at the third NTCIR workshop. In: Proceedings of the 3rd NTCIR workshop on research in information retrieval, automatic text summarization and question answering (NTCIR-3)

  7. Corrie B, Storey M-AD (2007) Toward understanding the importance of gesture in distributed scientific collaboration. Knowl Inform Syst 13(2): 143–171

    Article  Google Scholar 

  8. Coulouris G et al (1996) Distributed systems—concepts and design. Addison-Wesley, Reading

    Google Scholar 

  9. Crestani F, Wu S (2006) Testing the cluster hypothesis in distributed information retrieval. Inform Process Manage 42(5): 1137–1150

    Article  Google Scholar 

  10. de Kretser O et al (1998) Methodologies for distributed information retrieval. In: Proceedings of the 18th International Conference on Distributed Computing Systems, Amsterdam, pp 66–73

  11. Ellis D (1998) The dilemma of measurement in information retrieval research. J Am Soc Inform Sci 47(1): 23–36

    Article  Google Scholar 

  12. Fu L et al (2005) CQE: a collaborative querying environment. In: Marlino M, Sumner T, III FMS (eds) Proceedings of the 2005 ACM/IEEE Joint Conference on Digital Libraries (JCDL 2005). ACM Press, New York, p 378

  13. Gordon M, Kochen M (1989) Recall-precision trade-off: a derivation. J Am Soc Inform Sci 40(3): 145–151

    Article  Google Scholar 

  14. Gravano L, Garcia-Molina H (1995) Generalizing GlOSS to Vector-Space Databases and Broker Hierarchies. In: Proceedings of the 21th International Conference on Very Large Data Bases (VLDB ’95), San Francisco. Morgan Kaufmann, Menlo Park, pp 78–89

  15. Herrera-Viedma E et al (2004) Incorporating filtering techniques in a fuzzy linguistic multi-agent model for information gathering on the web. Fuzzy Sets Syst 148(1): 61–83

    Article  MATH  MathSciNet  Google Scholar 

  16. Huhns MN, Singh MP (1997) Agents on the web: agents are everywhere!. IEEE Internet Comput 1(1): 87

    Article  Google Scholar 

  17. Jung JJ (2007) Ontological framework based on contextual mediation for collaborative information retrieval. Inform Retr 10(1): 85–109

    Article  Google Scholar 

  18. Krichel T (2007) Information retrieval performance measures for a current awareness report composition aid. Inform Process Manage 43(4): 1030–1043

    Article  Google Scholar 

  19. Lee RST, Liu JNK (2004) iJADE Web-Miner: an intelligent agent framework for Internet shopping. IEEE Trans Knowl Data Eng 16(4): 461–473

    Article  MathSciNet  Google Scholar 

  20. Minkov E et al (2006) NER systems that suit user’s preferences: adjusting the recall-precision trade-off for entity extraction. In: Moore RC, Bilmes JA, Chu-Carroll J, Sanderson M (eds) Proceedings of the Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics

  21. Nguyen NT (2002) Consensus system for solving conflicts in distributed systems. Inform Sci 147(1–4): 91–122

    Article  MATH  MathSciNet  Google Scholar 

  22. Nguyen NT et al (2006) A consensus-based multi-agent approach for information retrieval in Internet. In: Alexandrov VN, van Albada GD, Sloot PMA, Dongarra J (eds) Proceedings of the 6th International Conference Computational Science (ICCS 2006). Lecture Notes in Computer Science, vol 3993. Springer, Berlin, pp 208–215

  23. Oates T et al (1997) Cooperative information-gathering: a distributed problem-solving approach. IEE Proc Softw 144(1): 72–88

    Article  Google Scholar 

  24. Palopoli L et al (2005) A graph-based approach for extracting terminological properties from information sources with heterogeneous formats. Knowl Inform Syst 8(4): 462–497

    Article  Google Scholar 

  25. Sakai T (2007) On the reliability of information retrieval metrics based on graded relevance. Inform Process Manage 43(2): 531–548

    Article  Google Scholar 

  26. Sandelin R (2007) Basic of Consensus. http://www.ic.org/nica/Process/Consensusbasics.htm

  27. Simeoni F et al (2007a) A grid-based infrastructure for distributed retrieval. In: Kovács L, Fuhr N, Meghini C (eds) Proceedings of the 11th European Conference on Digital Libraries (ECDL 2007), Budapest, Hungary, September 16–21. Lecture Notes in Computer Science, vol 4675. Springer, Berlin, pp 161–173

  28. Simeoni F et al (2007b) The DILIGENT framework for distributed information retrieval. In: Kraaij W, de Vries AP, Clarke CLA, Fuhr N, Kando N (eds) Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007), Amsterdam, July 23–27, pp 781–782

  29. Voorhees EM et al (1994) The collection fusion problem. In: TREC

  30. Wu S, Crestani F (2003) Distributed information retrieval: a multi-objective resource selection approach. Int J Uncertain Fuzziness Knowl Based Syst 11(1): 83–99

    Article  MATH  Google Scholar 

  31. Zudilova-Seinstra EV (2007) On the role of individual human abilities in the design of adaptive user interfaces for scientific problem solving environments. Knowl Inform Syst 13(2): 243–270

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Yeungnam University, Dae-Dong, Gyeonsan, South Korea

    Jason J. Jung

Authors
  1. Jason J. Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jason J. Jung.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jung, J.J. Consensus-based evaluation framework for distributed information retrieval systems. Knowl Inf Syst 18, 199–211 (2009). https://doi.org/10.1007/s10115-008-0153-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-008-0153-3

Keywords

Search

Navigation