Skip to main content
SpringerLink
Log in

On the evaluation of Geographic Information Retrieval systems

Evaluation framework and case study

  • Published:
International Journal on Digital Libraries Aims and scope Submit manuscript

Abstract

Search engines for Digital Libraries allow users to retrieve documents according to their contents. They process documents without differentiating the manifold aspects of information. Spatial and temporal dimensions are particularly dismissed. These dimensions are, however, of great interest for users of search engines targeting either the Web or specialized Digital Libraries. Recent studies reported that nearly 20% queries convey spatial and temporal information in addition to topical information. These three dimensions were referred to as parts of “geographic information.” In the literature, search engines handling those dimensions are called “Geographic Information Retrieval (GIR) systems.” Although several initiatives for evaluating GIR systems were undertaken, none was concerned with evaluating these three dimensions altogether. In this article, we address this issue by designing an evaluation framework, usefulness of which is highlighted through a case study involving a test collection and a GIR system. This framework allowed the comparison of our GIR system to state-of-the-art topical approaches. We also performed experiments for measuring performance improvement stemming from each dimension or their combination. We show that combining the three dimensions yields improvement in effectiveness (+73.9%) over a common topical baseline. Moreover, rather than conveying redundancy, the three dimensions complement each other.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Abbreviations

ACF:

Absolute calendar feature

AP:

Average precision

ASF:

Absolute spatial feature

CF:

Calendar feature

DCG:

Definite clause grammar

GIR:

Geographic information retrieval

GIS:

Geographic information system

IE:

Information extraction

IR:

Information retrieval

IV:

Information visualization

MAP:

Mean average precision

NDCG:

Normalized discounted cumulative gain

NER:

Named entity recognition

NLP:

Natural language processing

RCF:

Relative calendar feature

RSF:

Relative spatial feature

SF:

Spatial feature

TF:

Term frequency

TFc:

Continuous term frequency

References

  1. Purday J.: Think culture: Europeana.eu from concept to construction. Electron. Libr. 27(6), 919–937 (2009). doi:https://doi.org/10.1108/02640470911004039

    Article  Google Scholar 

  2. Usery E.L.: A feature-based geographic information system model. Photogramm. Eng. Rem. Sens. 62(7), 833–838 (1996) ISSN 0099-1112

    Google Scholar 

  3. Sanderson, M., Kohler, J.: Analyzing geographic queries. In: SIGIR-GIR’04: Proceedings of the Workshop on Geographic Information Retrieval at SIGIR (2004)

  4. Asadi, S., Chang, C.-Y., Zhou, X., Diederich, J.: Searching the World Wide Web for local services and facilities: a review on the patterns of location-based queries. In: Fan, W., Wu, Z., Yang, J. (eds.) WAIM’05: Proceedings of the 6th International Conference on Web-Age Information Management, LNCS, vol. 3739, pp. 91–101. Springer (2005). doi:https://doi.org/10.1007/11563952_9

  5. Souza, L.A., Davis, C.A., Jr., Borges, K.A.V., Delboni, T.M., Laender, A.H.F.: The role of gazetteers in geographic knowledge discovery on the Web. In: LA-WEB’05: Proceedings of the 3rd Latin American Web Congress, pp. 157–165. IEEE Computer Society, October 2005. doi:https://doi.org/10.1109/LAWEB.2005.38

  6. Gan, Q., Attenberg, J., Markowetz, A., Suel, T.: Analysis of geographic queries in a search engine log. In: LocWeb’08: Proceedings of the First International Workshop on Location and the Web, pp. 49–56. New York, NY, USA, 2008. ACM. doi:https://doi.org/10.1145/1367798.1367806

  7. Jones R., Zhang W.V., Rey B., Jhala P., Stipp E.: Geographic intention and modification in web search. Int. J. Geogr. Inf. Sci. 22(3), 229–246 (2008). doi:https://doi.org/10.1080/13658810701626186

    Article  Google Scholar 

  8. Gaio M., Sallaberry C., Etcheverry P., Marquesuzaa C., Lesbegueries J.: A global process to access documents’ contents from a geographical point of view. J. Vis. Lang. Comput. 19(1), 3–23 (2008). doi:https://doi.org/10.1016/j.jvlc.2007.08.010

    Article  Google Scholar 

  9. LeParc-Lacayrelle A., Gaio M., Sallaberry C.: La composante temps dans l’information géographique textuelle. Document Numérique 10(2), 129–148 (2007). doi:https://doi.org/10.3166/dn.10.2.129-148

    Article  Google Scholar 

  10. Sallaberry, C., Baziz, M., Lesbegueries, J., Gaio, M.: Towards an IE and IR system dealing with spatial information in digital libraries—evaluation case study. In: Cardoso, J., Cordeiro, J., Filipe, J. (eds.) ICEIS’07: Proceedings of the 9th International Conference on Enterprise Information Systems, pp. 190–197, 2007

  11. Manning, C.D., Raghavan, P., Schütze, H.: Introduction to information retrieval. Cambridge University Press, July 2008

  12. Gaizauskas R.J.: Recent advances in computational terminology edited by Didier Bourigault, Christian Jacquemin, and Marie-Claude L’Homme. Comput. Linguist. 29(2), 328–332 (2003). doi:https://doi.org/10.1162/coli.2003.29.2.328

    Article  Google Scholar 

  13. Cunningham, H., Maynard, D., Bontcheva, K., Tablan, V.: GATE: an architecture for development of robust HLT applications. In: ACL’02: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 168–175, Morristown, NJ, USA, 2002. ACL. doi:https://doi.org/10.3115/1073083.1073112

  14. Bilhaut, F., Widlöcher, A.: Linguastream: an integrated environment for computational linguistics experimentation. In: EACL’06: Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics: Posters & Demonstrations, pp. 95–98, Morristown, NJ, USA, 2006. ACL. doi: 10.1.1.134.4336

  15. Liu, Z., Gibbon, D.C., Shahraray, B.: Multimedia content acquisition and processing in the MIRACLE system. In: CCNC’06: Proceedings of the 3rd IEEE Conference on Consumer Communications and Networking, pp. 272–276, January 2006. doi:https://doi.org/10.1109/CCNC.2006.1593030

  16. Sagot B., Boullier P.: SxPipe 2: architecture pour le traitement pré syntaxique de corpus bruts. Traitement Automatique des Langues 49(2), 155–188 (2008)

    Google Scholar 

  17. Ferruci D., Lally A.: UIMA: an architectural approach to unstructured information processing in the corporate research environment. Nat. Lang. Eng., 10(3–4), 327–348 (2004). doi:https://doi.org/10.1017/S1351324904003523

    Article  Google Scholar 

  18. Spärck Jones, K.S.: A statistical interpretation of term specificity and its application in retrieval. J. Doc., 28(1), 11–21 (1972). ISSN 0022-0418. doi:https://doi.org/10.1108/eb026526

  19. Robertson S.E., Spärck Jones K.S.: Relevance weighting of search terms. J. Am. Soc. Inf. Sci. 27(3), 129–146 (1976). doi:https://doi.org/10.1002/asi.4630270302

    Article  Google Scholar 

  20. Salton G., Wong A., Yang C.S.: A vector space model for automatic indexing. Commun. ACM 18(11), 613–620 (1975). doi:https://doi.org/10.1145/361219.361220

    Article  Google Scholar 

  21. Larson, R.R., Frontiera, P.: Spatial ranking methods for geographic information retrieval (GIR) in digital libraries. In: ECDL’04: Proceedings of the 8th European Conference on Digital Libraries, LNCS, 3232, pp. 45–56. Springer, 2004. ISBN 3-540-23013-0. doi:https://doi.org/10.1007/978-3-540-30230-8_5

  22. Andogah, G.: Geographically Constrained Information Retrieval. PhD thesis, University of Groningen, Netherlands, May 2010

  23. Alonso, O.R.: Temporal information retrieval. PhD thesis, University of California, USA, May 2008

  24. Kalczynski, P.J., Chou, A.: Temporal document retrieval model for business news archives. Inf. Process. Manage, 41(3), 635–650, 2005. ISSN 0306-4573. doi:https://doi.org/10.1016/j.ipm.2004.01.002

    Article  Google Scholar 

  25. Woodruff, A.G., Plaunt, C.: Gipsy: automated geographic indexing of text documents. J. Am. Soc. Inf. Sci., 45(9): 645–655, 1994. ISSN 0002-8231. doi:https://doi.org/10.1002/(SICI)1097-4571(199410)45:9<645::AID-ASI2>3.0.CO;2-8

  26. Bilhaut, F., Charnois, T., Enjalbert, P., Mathet, Y.: Geographic reference analysis for geographic document querying. In: HLT- NAACL’03: Proceedings of the Workshop on Analysis of Geographic References, pp. 55–62, Morristown, NJ, USA, 2003. ACL. doi:https://doi.org/10.3115/1119394.1119403

  27. Vaid, S., Jones, C.B., Joho, H., Sanderson, M.: Spatio-textual indexing for geographical search on the Web. In: SSTD’05: Proceedings of the 9th International Symposium on Spatial and Temporal Databases, LNCS, vol. 3633, pp. 218–235. Springer, 2005. doi:https://doi.org/10.1007/11535331_13

  28. Valcartier. GRID—geospatial retrieval of indexed document. Technical report, R&D pour la défense, Canada (2006)

  29. Martins, B., Borbinha, J., Pedrosa, G., Gil, J., Freire, N.: Geographically-aware information retrieval for collections of digitized historical maps. In: GIR’07: Proceedings of the 4th ACM workshop on Geographical Information Retrieval, pp. 39–42, New York, NY, USA, 2007. ACM. doi:https://doi.org/10.1145/1316948.1316959

  30. Chen, Y.-F.R., Di Fabbrizio, G., Gibbon, D., Jora, S., Renger, B., Wei, B.: Geotracker: geospatial and temporal RSS navigation. In: WWW’07: Proceedings of the 16th International Conference on World Wide Web, pp. 41–50, New York, NY, USA, 2007. ACM. doi:https://doi.org/10.1145/1242572.1242579

  31. Lieberman, M.D., Samet, H., Sankaranarayanan, J., Sperling, J.: STEWARD: architecture of a Spatio-Textual search engine. In: GIS’07: Proceedings of the 15th Annual ACM International Symposium on Advances in Geographic Information Systems, pp. 1–8, New York, NY, USA, 2007. ACM. doi:https://doi.org/10.1145/1341012.1341045

  32. Pfoser, D., Efentakis, A., Hadzilacos, T., Karagiorgou, S., Vasiliou, G.: Providing universal access to history textbooks: a modified GIS case. In: W2GIS’09: Proceedings of the 9th International Symposium on Web and Wireless Geographical Information Systems, LNCS, vol. 5886, pp. 87–102, 2009. doi:https://doi.org/10.1007/978-3-642-10601-9_7

  33. Buscaldi, D., Rosso, P.: Geooreka: enhancing web searches with geographical information. In: De Antonellis, V., Castano, S., Catania, B., Guerrini, G. (eds.) SEBD’09: Proceedings of the Seventeenth Italian Symposium on Advanced Database Systems, pp. 205–212. Edizioni Seneca, 2009

  34. García-Cumbreras, M.Á., Perea-Ortega, J.M., García-Vega, M., Ureña-López, L.A.: Information retrieval with geographical references. Relevant documents filtering vs. query expansion. Inf. Process. Manage. 45(5), 605–614 (2009). ISSN 0306-4573. doi:https://doi.org/10.1016/j.ipm.2009.04.006

  35. Strötgen, J., Gertz, M., Popov, P.: Extraction and exploration of spatio-temporal information in documents. In: GIR’10: Proceedings of the 6th Workshop on Geographic Information Retrieval, pp. 16:1–16:8, New York, NY, USA, 2010. ACM. doi:https://doi.org/10.1145/1722080.1722101

  36. Brisaboa N., Luaces M., Places , Á. , Seco D.: Exploiting geographic references of documents in a geographical information retrieval system using an ontology-based index. GeoInformatica 14(3), 307–331 (2010). doi:https://doi.org/10.1007/s10707-010-0106-3

    Article  Google Scholar 

  37. Stokes, N., Li, Y., Moffat, A., Rong, J. An empirical study of the effects of NLP components on Geographic IR performance. Int. J. Geogr. Inf. Sci. 22(3), 247–264, 2008. ISSN 1365-8816. doi:https://doi.org/10.1080/13658810701626210

  38. Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., Lioma, C.: Terrier: a high performance and scalable information retrieval platform. In: OSIR’06: Proceedings of ACM SIGIR’06 Workshop on Open Source Information Retrieval (2006)

  39. Clough, P., Joho, H., Purves, R.: Judging the spatial relevance of documents for GIR. In: ECIR’06: Proceedings of the 28th European Conference on IR Research, LNCS, vol. 3936, pp. 548–552. Springer, 2006. doi:https://doi.org/10.1007/11735106_62

  40. Martins, B., Silva, M.J., Andrade, L.: Indexing and ranking in Geo-IR systems. In: GIR’05: Proceedings of the 2005 Workshop on Geographic Information Retrieval, pp. 31–34, New York, NY, USA, 2005. ACM. doi:https://doi.org/10.1145/1096985.1096993

  41. Jones, C.B., Purves, R.: GIR’05 2005 ACM workshop on geographical information retrieval. SIGIR Forum, 40(1), 34–37 (2006). ISSN 0163-5840. doi:https://doi.org/10.1145/1147197.1147202

  42. Larson, R.R.: Geographic information retrieval and digital libraries. In: ECDL’09: Proceedings of the 13th European Conference on Digital Libraries, LNCS, vol. 5714, pp. 461–464. Springer, 2009. doi:https://doi.org/10.1007/978-3-642-04346-8_59

  43. Widlöcher, A., Bilhaut, F.: La plate-forme LinguaStream: un outil d’exploration linguistique sur corpus. In: TALN’05: Actes de la 12e Conférence sur le Traitement Automatique du Langage Naturel (2005)

  44. Kergosien, E., Kamel, M., Sallaberry, C., Bessagnet, M.-N., Aussenac-Gilles, N., Gaio, M.: Construction automatique d’ontologie et enrichissement à partir de ressources externes. In: JFO’09: Actes des 3e Journées Francophones sur les Ontologies, pp. 11–20, 2009

  45. Sallaberry, C., Gaio, M., Palacio, D., Lesbegueries, J.: Fuzzying GIS topological functions for GIR needs. In: GIR’08: Proceeding of the 2nd international Workshop on Geographic Information Retrieval, pp. 1–8, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-253-5. doi:https://doi.org/10.1145/1460007.1460008

  46. Chrisman N.R.: Deficiencies of sheets and tiles: building sheetless databases. Int. J. Geogr. Inf. Sci. 4(2), 157–167 (1990). doi:https://doi.org/10.1080/02693799008941537

    Article  Google Scholar 

  47. Palacio, D., Sallaberry, C., Gaio, M.: Normalizing spatial information to improve geographical information indexing and retrieval in digital libraries. In: ISGIS’10: Proceedings of the Joint International Conference on Theory, Data Handling and Modelling in GeoSpatial Information Science proceedings, pp. 229–234 (2010)

  48. Palacio, D., Sallaberry, C., Gaio, M.: Normalizing spatial information to better combine criteria in geographical information retrieval. In: ECIR-GIIW’09: Proceeding of the International Workshop on Geographic Information on the Internet, pp. 37–48, 2009

  49. Fox, E.A., Shaw, J.A.: Combination of multiple searches. In: Harman, D.K. (ed.) TREC-1: Proceedings of the First Text REtrieval Conference, pp. 243–252, Gaithersburg, MD, USA, February 1993. NIST

  50. Hubert G., Mothe J.: An adaptable search engine for multimodal information retrieval. J. Am. Soc. Inf. Sci. Technol., 60(8), 1625–1634 (2009). doi:https://doi.org/10.1002/asi.21091

    Article  Google Scholar 

  51. Lee, J.H.: Analyses of multiple evidence combination. In: SIGIR’97: Proceedings of the 20th Annual International ACM SIGIR Conference, pp. 267–276, New York, NY, USA, 1997. ACM Press. doi:https://doi.org/10.1145/258525.258587

  52. de Borda, J.-C.: Mémoire sur les élections au scrutin. In: Histoire de l’Académie Royale des Sciences, Paris (1781)

  53. de Condorcet, M.: Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. Imprimerie Royale, Paris (1785)

  54. Saari, D.: Which is better: the Condorcet or Borda winner? Soc. Choice Welfare 26(1), 107–129, 2006. ISSN 0176-1714. doi:https://doi.org/10.1007/s00355-005-0046-2

  55. Aslam, J.A., Montague, M.: Models for metasearch. In: SIGIR’01: Proceedings of the 24th Annual International ACM SIGIR Conference, pp. 276–284, New York, NY, USA, 2001. ACM. doi:https://doi.org/10.1145/383952.384007

  56. Montague, M., Aslam, J.A.: Condorcet fusion for improved retrieval. In: CIKM’02: Proceedings of the 11th International Conference on Information and Knowledge Management, pp. 538–548, New York, NY, USA, 2002. ACM. doi:https://doi.org/10.1145/584792.584881

  57. Cabanac, G., Hubert, G., Boughanem, M., Chrisment, C.: Tie-breaking Bias: Effect of an Uncontrolled Parameter on Information Retrieval Evaluation. In: Agosti, M., Ferro, N., Peters, C., de Rijke, M., Smeaton, A.F. (eds.) CLEF’10: Proceedings of the 1st Conference on Multilingual and Multimodal Information Access Evaluation, LNCS, vol. 6360, pp. 112–123. Springer, 2010. doi:https://doi.org/10.1007/978-3-642-15998-5_13

  58. Farah, M., Vanderpooten, D.: An outranking approach for rank aggregation in information retrieval. In: SIGIR’07: Proceedings of the 30th Annual International ACM SIGIR Conference, pp. 591–598, New York, NY, USA, 2007. ACM. doi:https://doi.org/10.1145/1277741.1277843

  59. Liu, B.: Information Retrieval and Web Search. In: Web Data Mining, Data-Centric Systems and Applications, chapter 6, pp. 183–236. Springer, 2007. ISBN 978-3-540-37882-2. doi:https://doi.org/10.1007/978-3-540-37882-2_6

  60. Cleverdon, C.W.: Report on the testing and analysis of an investigation into the comparative efficiency of indexing systems. ASLIB Cranfield Research Project, ASLIB, Cranfield, UK, October 1962

  61. Robertson, S.: On the history of evaluation in IR. J. Inf. Sci. 34(4), 439–456, 2008. doi:https://doi.org/10.1177/0165551507086989

    Article  Google Scholar 

  62. Sanderson, M.: Test collection based evaluation of information retrieval systems. Found. Trends Inf. Retr. 4(4):247–375, 2010. ISSN 1554-0669. doi:https://doi.org/10.1561/1500000009

  63. Harman, D.K.: The TREC test collections. In: Voorhees and Harman [67], chapter 2, pp. 21–53

  64. Buckley, C., Voorhees, E.M.: Evaluating evaluation measure stability. In: SIGIR’00: Proceedings of the 23rd International ACM SIGIR Conference, pp. 33–40, New York, NY, USA, 2000. ACM. doi:https://doi.org/10.1145/345508.345543

  65. Voorhees, E.M.: The philosophy of information retrieval evaluation. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) CLEF’01: Proceedings of the Second Workshop of the Cross-Language Evaluation Forum, LNCS, vol. 2406, pp. 355–370. Springer, 2002. doi:https://doi.org/10.1007/3-540-45691-0_34

  66. Harman, D.K. (ed.): TREC-1: Proceedings of the First Text REtrieval Conference, Gaithersburg, MD, USA, February 1993. NIST

  67. Voorhees, E.M., Harman, D.K.: TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge, MA, USA, 2005

  68. Kando, N., Kuriyama, K., Nozue, T., Eguchi, K., Kato, H., Hidaka, S.: Overview of IR Tasks at the First NTCIR Workshop. In: Proceedings of the First NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition, pp. 11–44. NACSIS, 1999

  69. Kando, N.: Evaluation of information access technologies at the NTCIR workshop. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) Comparative Evaluation of Multilingual Information Access Systems, LNCS, vol. 3237, pp. 197–221. Springer, 2004. doi:https://doi.org/10.1007/978-3-540-30222-3_4

  70. Peters, C. (ed.): CLEF’01: Proceedings of the 2nd Workshop of the Cross-Language Evaluation Forum, LNCS, vol. 2069. Springer, 2001. doi:https://doi.org/10.1007/3-540-44645-1

  71. Peters C., Braschler M.: European Research Letter—Cross- Language System Evaluation: the CLEF Campaigns. J. Am. Soc. Inf. Sci. Technol. 52(12), 1067–1072 (2001). doi:https://doi.org/10.1002/asi.1164

    Article  Google Scholar 

  72. Fuhr, N., Gövert, N., Kazai, G., Lalmas, M. (eds.): INEX’02: Proceedings of the First Workshop of the INitiative for the Evaluation of XML Retrieval (INEX), 2002

  73. Kazai G., Lalmas M., Fuhr N., Fuhr N., Fuhr N.: A report on the first year of the initiative for the evaluation of XML retrieval. J. Am. Soc. Inf. Sci. Technol. 55(6), 551–556 (2004). doi:https://doi.org/10.1002/asi.10386

    Article  Google Scholar 

  74. Verhagen M., Gaizauskas R., Schilder F., Hepple M., Moszkowicz J., Pustejovsky J.: The TempEval challenge: identifying temporal relations in text. Lang. Resour. Eval. 43(2), 161–179 (2009). doi:https://doi.org/10.1007/s10579-009-9086-z

    Article  Google Scholar 

  75. Bucher, B., Clough, P., Joho, H., Purves, R., Syed, A.K.: Geographic IR systems: Requirements and evaluation. In: ICC’05: Proceedings of the 22nd International Cartographic Conference. Global Congressos, 2005. CDROM

  76. Gey, F.C., Larson, R.R., Sanderson, M., Joho, H., Clough, P., Petras, V.: GeoCLEF’05: the CLEF 2005 cross-language geographic information retrieval track overview. In: CLEF’05: Proceedings of the 5th Workshop on Cross-Language Evalution Forum, LNCS, vol. 4022, pp. 908–919. Springer, 2006. doi:https://doi.org/10.1007/11878773_101

  77. Perea-Ortega, J.M., García-Cumbreras, M.A., García-Vega, M., Ureña-López, L. A. Comparing several textual information retrieval systems for the geographical information retrieval task. In: NLDB’08: Proceedings of the 13th International Conference on Natural Language and Information Systems, pp. 142–147, Berlin, Heidelberg, 2008. Springer-Verlag. doi:https://doi.org/10.1007/978-3-540-69858-6_15

  78. Ogilvie, P., Callan, J.P.: Experiments using the Lemur toolkit. In: TREC’01: Proceedings of the 9th Text REtrieval Conference, Gaithersburg, MD, USA, February 2001. NIST

  79. Gospodnetić, O., Hatcher, E.: Lucene in Action. Manning Publications (2005)

  80. Gey, F., Larson, R., Kando, N., Machado, J., Sakai, T.: NTCIR-GeoTime Overview: Evaluating Geographic and Temporal Search. In: NTCIR’10: Proceedings of the 8th NTCIR Workshop, pp. 147–153, Tockyo, Japan, 2010. NII

  81. Santos, D., Cabral, L.: GikiCLEF: Expectations and Lessons Learned. In: Peters, C., Di Nunzio, G., Kurimo, M., Mostefa, D., Penas, A., Roda, G. (eds.) CLEF’09: Proceedings of the 9th Workshop of the Cross-Language Evaluation Forum, LNCS, vol. 6241, pp. 212–222. Springer Berlin/Heidelberg, 2010. doi:https://doi.org/10.1007/978-3-642-15754-7_23

  82. Palacio, D., Cabanac, G., Sallaberry, C., Hubert, G.: Measuring effectiveness of geographic IR systems in digital libraries: evaluation framework and case study. In: ECDL’10: Proceedings of the 14th European Conference on Research and Advanced Technology for Digital Libraries, LNCS, vol. 6273, pp. 340–351. Springer, 2010. doi:https://doi.org/10.1007/978-3-642-15464-5_34

  83. Järvelin K., Kekäläinen J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. 20(4), 422–446 (2002). doi:https://doi.org/10.1145/582415.582418

    Article  Google Scholar 

  84. Hull, D.: Using statistical testing in the evaluation of retrieval experiments. In: SIGIR’93: Proceedings of the 16th Annual International ACM SIGIR Conference, pp. 329–338, New York, NY, USA, 1993. ACM Press. doi:https://doi.org/10.1145/160688.160758

  85. da Costa Pereira, C., Dragoni, M., Pasi, G.: Multidimensional relevance: A new aggregation criterion. In: ECIR’09: Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval, pp. 264–275, Berlin, Heidelberg (2009). Springer-Verlag. doi:https://doi.org/10.1007/978-3-642-00958-7_25

  86. Farah M., Vanderpooten D.: An outranking approach for information retrieval. Inf. Retr. 11(4), 315–334 (2008). doi:https://doi.org/10.1007/s10791-008-9046-z

    Article  Google Scholar 

  87. Andogah, G., Bouma, G.: Relevance measures using geographic scopes and types. In: CLEF’07: Proceedings of the 7th Workshop of the Cross-Language Evaluation Forum, LNCS, vol. 5152, pp. 794–801 (2008). doi:https://doi.org/10.1007/978-3-540-85760-0_100

  88. Yager R.R.: On ordered weighted averaging aggregation operators in multicriteria decisionmaking. IEEE Trans. Syst. Man Cybern. 18(1), 183–190 (1988). doi:https://doi.org/10.1109/21.87068

    Article  MathSciNet  Google Scholar 

  89. Boughanem, M., Loiseau, Y., Prade, H.: Refining aggregation functions for improving document ranking in information retrieval. In: SUM’07: Proceedings of the 1st International Conference on Scalable Uncertainty Management, pp. 255–267, Berlin, Heidelberg, 2007. Springer. doi:https://doi.org/10.1007/978-3-540-75410-7_19

  90. Labreuche C., Grabisch M.: The Choquet integral for the aggregation of interval scales in multicriteria decision making. Fuzzy Sets Syst. 137(1), 11–26 (2003). doi:https://doi.org/10.1016/S0165-0114(02)00429-3

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Université de Pau et des Pays de l’Adour, LIUPPA ÉA 3000, Avenue de l’Université, BP 1155, 64013, Pau cedex, France

    Damien Palacio & Christian Sallaberry

  2. Université de Toulouse, IRIT UMR 5505 CNRS, 118 route de Narbonne, 31062, Toulouse cedex 9, France

    Guillaume Cabanac & Gilles Hubert

Authors
  1. Damien Palacio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guillaume Cabanac
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christian Sallaberry
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gilles Hubert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guillaume Cabanac.

Additional information

This paper is a substantially revised and extended version of a paper (with the title “Measuring Effectiveness of Geographic IR Systems in Digital Libraries: Evaluation Framework and Case Study”) originally appeared in the Proceedings of the 14th European Conference on Digital Libraries (ECDL 2010).

Electronic Supplementary Material

The Below is the Electronic Supplementary Material.

ESM 1 (ZIP 3511 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Palacio, D., Cabanac, G., Sallaberry, C. et al. On the evaluation of Geographic Information Retrieval systems. Int J Digit Libr 11, 91–109 (2010). https://doi.org/10.1007/s00799-011-0070-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00799-011-0070-z

Keywords

Search

Navigation