Skip to main content
Springer Nature Link
Log in

Managing the Knowledge Creation Process of Large-Scale Evaluation Campaigns

  • Conference paper
Research and Advanced Technology for Digital Libraries (ECDL 2009)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 5714))

Included in the following conference series:

Abstract

This paper discusses the evolution of large-scale evaluation campaigns and the corresponding evaluation infrastructures needed to carry them out. We present the next challenges for these initiatives and show how digital library systems can play a relevant role in supporting the research conducted in these fora by acting as virtual research environments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Ackoff, R.L.: From Data to Wisdom. Journal of Applied Systems Analysis 16, 3–9 (1989)

    Google Scholar 

  2. Agosti, M., Di Nunzio, G.M., Ferro, N.: A Proposal to Extend and Enrich the Scientific Data Curation of Evaluation Campaigns. In: Proc. 1st International Workshop on Evaluating Information Access (EVIA 2007), pp. 62–73. National Institute of Informatics, Tokyo (2007)

    Google Scholar 

  3. Agosti, M., Di Nunzio, G.M., Ferro, N., Harman, D., Peters, C.: The Future of Large-scale Evaluation Campaigns for Information Retrieval in Europe. In: Kovács, L., Fuhr, N., Meghini, C. (eds.) ECDL 2007. LNCS, vol. 4675, pp. 509–512. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  4. Agosti, M., Ferro, N.: A Formal Model of Annotations of Digital Content. ACM Transactions on Information Systems (TOIS) 26(1), 3:1–3:57 (2008)

    Google Scholar 

  5. Agosti, M., Ferro, N.: Towards an Evaluation Infrastructure for DL Performance Evaluation. In: Evaluation of Digital Libraries: An Insight to Useful Applications and Methods. Chandos Publishing (2009)

    Google Scholar 

  6. Cleverdon, C.W.: The Cranfield Tests on Index Languages Devices. In: Readings in Information Retrieval, pp. 47–60. Morgan Kaufmann Publisher, San Francisco (1997)

    Google Scholar 

  7. Di Nunzio, G.M., Ferro, N.: DIRECT: a System for Evaluating Information Access Components of Digital Libraries. In: Rauber, A., Christodoulakis, S., Tjoa, A.M. (eds.) ECDL 2005. LNCS, vol. 3652, pp. 483–484. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  8. Dussin, M., Ferro, N.: Design of a Digital Library System for Large-Scale Evaluation Campaigns. In: Christensen-Dalsgaard, B., Castelli, D., Ammitzbøll Jurik, B., Lippincott, J. (eds.) ECDL 2008. LNCS, vol. 5173, pp. 400–401. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  9. Dussin, M., Ferro, N.: The Role of the DIKW Hierarchy in the Design of a Digital Library System for the Scientific Data of Large-Scale Evaluation Campaigns. In: Proc. 8th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2008), p. 450. ACM Press, New York (2008)

    Google Scholar 

  10. European Commission Information Society and Media. i2010: Digital Libraries (October 2006), http://europa.eu.int/information_society/activities/digital_libraries/doc/brochures/dl_brochure_2006.pdf

  11. Ferro, N.: Digital Annotations: a Formal Model and its Applications. In: Information Access through Search Engines and Digital Libraries, pp. 113–146. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  12. Ferro, N., Peters, C.: From CLEF to TrebleCLEF: the Evolution of the Cross-Language Evaluation Forum. In: Proc. 7th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Cross-Lingual Information Access, pp. 577–593. National Institute of Informatics, Tokyo (2008)

    Google Scholar 

  13. Fricke, M.: The Knowledge Pyramid: a Critique of the DIKW Hierarchy. Journal of Information Science 35(2), 131–142 (2009)

    Article  Google Scholar 

  14. Fuhr, N., et al.: Evaluation of Digital Libraries. International Journal on Digital Libraries, 8(1):21–38 (2007)

    Google Scholar 

  15. Harman, D.K., Voorhess, E.M. (eds.): TREC. Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge (2005)

    Google Scholar 

  16. National Science Board. Long-Lived Digital Data Collections: Enabling Research and Education in the 21st Century (NSB-05-40). National Science Foundation (NSF) (September 2005), http://www.nsf.gov/pubs/2005/nsb0540/

  17. Nonaka, I., Takeuchi, H.: The knowledge-creating company: How Japanese companies create the dynamics of innovation. Oxford University Press, USA (1995)

    Google Scholar 

  18. Rowley, J.: The Wisdom Hierarchy: Representations of the DIKW Hierarchy. Journal of Information Science 33(2), 163–180 (2007)

    Article  Google Scholar 

  19. Working Group on Data for Science. FROM DATA TO WISDOM: Pathways to Successful Data Management for Australian Science. Report to the Primw Minister’s Science, Engineering and Innovation Council (PMSEIC) (September 2006), http://www.dest.gov.au/sectors/science_innovation/publications_resources/profiles/Presentation_Data_for_Science.htm

  20. Zeleny, M.: Management Support Systems: Towards Integrated Knowledge Management. Human Systems Management 7(1), 59–70 (1987)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Padua, Italy

    Marco Dussin & Nicola Ferro

Authors
  1. Marco Dussin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicola Ferro
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Information Engineering, University of Padua, Via Gradenigo 6/a, 35131, Padova, Italy

    Maristella Agosti

  2. Department of Computer Science and Engineering IST, Instituto Superior Técnico, Av. Rovisco Pais, 1049-001, Lisboa, Portugal

    José Borbinha

  3. Department of Archives and Library Sciences, Ionian University, 72 Ioannou Theotoki str., 49100, Corfu, Greece

    Sarantos Kapidakis

  4. Department of Archives and Library Sciences, Ionian University, 72 Ioannou Theotoiki str., 49100, Corfu, Greece

    Christos Papatheodorou  & Giannis Tsakonas  & 

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Dussin, M., Ferro, N. (2009). Managing the Knowledge Creation Process of Large-Scale Evaluation Campaigns. In: Agosti, M., Borbinha, J., Kapidakis, S., Papatheodorou, C., Tsakonas, G. (eds) Research and Advanced Technology for Digital Libraries. ECDL 2009. Lecture Notes in Computer Science, vol 5714. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04346-8_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-04346-8_8

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-04345-1

  • Online ISBN: 978-3-642-04346-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics