Skip to main content
SpringerLink
Log in

Evaluation of digital libraries

  • Regular Paper
  • Published:
International Journal on Digital Libraries Aims and scope Submit manuscript

Abstract

Digital libraries (DLs) are new and innovative information systems, under constant development and change, and therefore evaluation is of critical importance to ensure not only their correct evolution but also their acceptance by the user and application communities. The Evaluation activity of the DELOS Network of Excellence has performed a large-scale survey of current DL evaluation activities. This study has resulted in a description of the state of the art in the field, which is presented in this paper. The paper also proposes a new framework for the evaluation of DLs, as well as for recording, describing and analyzing the related research field. The framework includes a methodology for the classification of current evaluation procedures. The objective is to provide a set of flexible and adaptable guidelines for DL evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Fuhr, N., Hansen, P., Mabe, M., Micsik, A., Solvberg, I.: Digital Libraries: A Generic Classification and Evaluation Scheme. In: Lecture Notes In Computer Science, vol. 2163, pp. 187–199. Springer, Berlin (2001) Research and Advanced Technology for Digital Libraries: 5th European Conference, ECDL 2001, Darmstadt, Germany, September 4–9, 2001. Proceedings

  2. DELOS: DELOS Workshop on Evaluation of Digital Libraries: Testbeds, Measurements, and Metrics. Technical report (2002) http://www.sztaki.hu/conferences/deval

  3. Agosti, M., Fuhr, N. (eds.) Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries, http://dlib.ionio.gr/wp7/workshop2004_program.html (2004).

  4. Cleverdon, C.: The Cranfield tests on index language devices. In: Sparck-Jones, K., Willett, P. (eds.) Readings in Information Retrieval, pp. 47–59 Morgan Kaufmann, San Francisco (1997)

  5. Turpin, A., Hersh, W.: Why batch and user evaluations do not give the same results. In: Proceedings of the SIGIR01, pp. 225–231. ACM, New York (2001)

  6. CCSDS: A reference model for an open archival information system. Document Number: ISO 14721:2003 (2003) http://ssdoo.gsfc.nasa.gov/nost/wwwclassic/documents/pdf/ CCSDS-650.0-B- 1.pdf

  7. IFLANET: Functional requirements for bibliographic records (1998) http://www.ifla.org/VII/s13/frbr/frbr.pdf

  8. Levy D. and Marshall C. (1995). Going digital: a look at assumptions underlying digital libraries.. Commun. ACM 38: 77–84

    Article  Google Scholar 

  9. Borgman, C.: What are digital libraries? Competing visions. Inf. Process. Manage. 35, 277–243 (1999)

    Google Scholar 

  10. DELOS: Reference models for digital libraries: actors and roles - final report. Technical report (2003) http://www.dli2. nsf.gov/internationalprojects/working_group_reports/ actors_final_report.html

  11. Agosti, M., Candela, L., Castelli, D., Ferro, N., Ioannidis Y., Koutrika, G., Meghini, C., Pagano, P., Ross, S., Schek, H.J., Schuldt, H.: A reference model for DLMSs interim report. In: Candela, L., Castelli, D. (eds.) Deliverable D1.4.2 – Reference Model for Digital Library Management Systems [Draft 1], DELOS, A Network of Excellence on Digital Libraries – IST-2002-2.3.1.12, Technology-enhanced Learning and Access to Cultural Heritage – http://146.48.87.122:8003/OLP/Repository/1.0/Disseminate/ delos/2006_WP1_D142/content/pdf?version=1 [last visited 2006, October 2] (2006)

  12. Goncalves M.A., Fox E., Kipp N. and Watson L. (2004). Streams, structures, spaces, scenarios, societies (5S): a formal model for digital libraries. ACM Trans. Inf. Syst. 22: 270–312

    Article  Google Scholar 

  13. Baker, S., Lancaster, F.: The measurement and evaluation of library services. Information Resources Press, Arlington (1991)

  14. Nicholson S. (2004). A conceptual framework for the holistic measurement and cumulative evaluation of library services. J. Doc. 60: 164–182

    Article  Google Scholar 

  15. Kyrillidou, M., Lorie, M., Hobbs, B., Choudhury, G., Webster, D., Flores, N., Heath, F.: Emerging tools for evaluating DL services: conceptual adaptations of LIBQUAL+ and CAPM. J. Digit. Inf. 4, (2003) http://jodi.ecs.soton.ac.uk/ Articles/v04/i02/Heath

  16. Kyrillidou, M., Giersch, S.: Developing the digiqual protocol for digital library evaluation. In: JCDL ’05: Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries, New York, NY, USA, pp. 172–173. ACM, New York (2005)

  17. Tsakonas, G., Kapidakis, S., Papatheodorou, C.: Evaluation of user interaction in digital libraries. In: Agosti, M., Fuhr, N. (eds.) Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries, Padua, Italy (2004) http://dlib. ionio.gr/wp7/workshop2004_program.html

  18. Tsakonas, G., Papatheodorou, C.: Analyzing and evaluating usefulness and usability in electronic information services. J. Inf. Sci. 32(5), 400–419 (2006)

    Google Scholar 

  19. Marchionini, G.: Evaluating digital libraries: a longitudinal and multifaceted view. Library Trends 49, (2000)

  20. DELOS: A Network of Excellence on Digital Libraries (2004) Technical Annex 1, pag. 6 - DELOS Network of Excellence on Digital Libraries, as part of the Information Society Technologies (IST) Program of the European Commission (Contract G038-507618)

  21. ISO: ISO 9241-11. Ergonomic Requirements for Office Work With Visual Display Terminals (VDT). Part 11: Guidance in Usability. International Standards Organization, London (1997)

  22. DeGroot, S.P., Knapp, A.E.: Applying the user-centered design (ucd) process to the development of a large bibliographic navigation tool: a partnership between librarian, researcher and developer. Scopus White Paper Series (2004) http://www.elsevier.com/librarians

  23. Blandford A., Buchanan G. and Jones M. (2004). Usability of digital libraries. Int. J. Digit. Libr. 4: 69–70

    Article  Google Scholar 

  24. Dickstein, R., Mills, V.: Usability testing at the University of Arizona Library: how to let the users in on the design. Inf. Technol. Libr. 19, (2000)

  25. Hilbert D.M. and Redmiles D.F. (2000). Extracting usability information from user interface events. ACM Comput. Surv. 32: 384–421

    Article  Google Scholar 

  26. Ivory M. and Hearst M. (2001). The state of the art in automating usability evaluation of user interfaces. ACM Comput. Surv. 33: 470–516

    Article  Google Scholar 

  27. Ke H.R., Kwakkelaar R., Tai Y.M. and Chen L.C. (2002). Exploring behavior of E-journal users in science and technology: transaction log analysis of Elsevier’s ScienceDirect OnSite in Taiwan. Libr. Inf. Sci. Res. 24: 265–291

    Article  Google Scholar 

  28. Sfakakis, M., Kapidakis, S.: User behavior tendencies on data collections in a digital library. In: Lecture Notes in Computer Science, vol. 2458, pp. 550—559. Springer, Berlin (2002) Research and Advanced Technology for Digital Libraries: 6th European Conference, ECDL 2002, Rome, Italy, September 16–18, 2002. Proceedings

  29. Jones S., Cunningham S.J., McNab R. and Boddie S. (2000). A transaction log analysis of a digital library. Int. J. Digit. Libr. 3: 152–169

    Article  Google Scholar 

  30. Sutcliffe A.G., Ennis M. and Hu J. (2000). Evaluating the effectiveness of visual user interfaces for information retrieval. Int. J. Human-Comput. Stud. 53: 741–763

    Article  MATH  Google Scholar 

  31. Kengeri R., Fox E.A., Reddy H.P., Harley H.D. and Seals C.D. (1999). Usability study of digital libraries: ACM, IEEE-CS NCSTRL, NDLTD. Int. J. Digit. Libr. 2: 157–169

    Article  Google Scholar 

  32. Clark J.A. (2004). A usability study of the Belgian American Research Collection: measuring the functionality of a digital library. OCLC Syst. Serv. Int. Digit. Libr. Perspect. 20: 115–127

    Google Scholar 

  33. Rousseau G.K., Rogers W., Mead S.E., Sit R.A. and Rousseau J.B.A. (1998). Assessing the usability of on-line library systems. Behav. Inf. Technol. 17: 274–281

    Article  Google Scholar 

  34. Thong J.Y.L., Hong W. and Tam K. (2002). Understanding user acceptance of digital libraries: what are the roles of interface characteristics, organizational context, and individual differences? Int. J. Human-Comput. Stud. 57: 215–242

    Article  Google Scholar 

  35. Hartson R.H., Shivakumar P. and Perez-Quinones M.A. (2004). Usability inspection of digital libraries: a case study. Int. J. Digit. Libr. 4: 108–123

    Article  Google Scholar 

  36. Glosiene, A., Manzuch, Z.: Usability of ICT-based Systems: State-of-the-art Review. Technical Report CALIMERA Deliverable 9 (2004)

  37. Dicks, R.S.: Mis-usability: on the uses and misuses of usability testing. In: Proceedings of the 20th Annual International Conference on Computer Documentation, New York, pp. 26–30 ACM, New York (2002)

  38. Doubleday, A., Springett, M.V., Sutcliffe, A.G., Ryan, M.: A comparison of usability techniques for evaluating design. In: Proceedings of the Conference on “Designing Interactive Systems: Processes, Practices, Methods and Techniques”, New York, pp. 101–110 ACM, New York (1997)

  39. Hill L.L., Carver L., Smith T.R., Frew J., Larsgaard M., Dolin R. and Rae M.A. (2000). Alexandria digital library: user evaluation studies and system design. J. Am. Soc. Inf. Sci. 51: 246–259

    Article  Google Scholar 

  40. Bishop A.P., Merkel C., Neumann L.J., Star S.L., Sandusky R.J. and Ignacio E. (2002). Digital libraries: situating use in changing information infrastructure. J. Am. Soc. Inf. Sci. 51: 394–413

    Article  Google Scholar 

  41. Fox E.A., Heath L.S., Rao D., Brueni D.J., Nowell L.T., Wake W.C. and Hix D. (1993). Users, user interfaces, and objects: envision, a digital library. J. Am. Soc. Inf. Sci. 44: 480–491

    Article  Google Scholar 

  42. Buttenfield, B.: Digital libraries: philosophies, technical design considerations, and example scenarios. In: Usability Evaluation of Digital Libraries, pp. 39–59. Haworth Press, New York (1999)

  43. Blandford, A., Stelmaszewska, H., Bryann-Kinns, N.: Use of multiple digital libraries: a case study. In: Proceedings of the first ACM/IEEE-CS joint conference on digital libraries, pp. 179—188. ACM Press (2001)

  44. Castillo, J., Hartson, H.R., Hix, D.: Remote usability evaluation: can users report their own critical incidents. In: CHI 1998 Conference Summary on Human Factors in Computing Systems, New York, pp. 253–254 ACM, New York (1998)

  45. Bates M. (1990). Where should the person stop and the information search interface start? Inf. Process. Manage. 26: 575–591

    Article  Google Scholar 

  46. Cousins, S.B.: A task-oriented interface to a DL. In: Tauber, M.J. (ed.) Conference Companion on Human Factors in Computing Systems: Common Ground, New York, pp. 103–104. ACM, New York (1996)

  47. Pejtersen, A., Fidel, R.: A framework for work centered evaluation and design: a case study of IR on the web. Technical report (1998)

  48. Rasmussen, J., Pejtersen, A., Goodstein, L.: Cognitive Systems Engineering. Whiley, New York (1994)

  49. Borgman, C.: From Gutenberg to the global information infrastructure: Access to information in the networked World. MIT, Cambridge (2000)

  50. Tenopir, C.: Use and users of electronic library resources: an overview and analysis of recent research studies. Council on Library and Information Resources, Washington (2003)

  51. Banwell L., Ray K., Coulson G., Urquhart C., Lonsdale R., Armstrong C., Thomas R., Spink S., Yeoman A., Fenton R. and Rowley J. (2004). The JISC user behaviour monitoring and evaluation framework. J. Document. 60: 302–320

    Article  Google Scholar 

  52. Payette S.D. and Rieger O.Y. (1998). Supporting scholarly inquiry: incorporating users in the design of the digital library. J. Acad. Libr. 24: 121–129

    Article  Google Scholar 

  53. Yu L. and Apps A. (2000). Studying E-journal user behavior using log files: the experience of superjournal. Libr. Inf. Sci. Res. 22: 311–338

    Article  Google Scholar 

  54. Wilson T.D. (1997). Information behaviour: an interdisciplinary perspective. Inf. Process. Manage. 33: 551–572

    Article  Google Scholar 

  55. Kuhlthau C.C. (1991). Inside the search process: information seeking from the user’s perspective. J. Am. Soc. Inf. Sci. 42: 361–371

    Article  Google Scholar 

  56. Ellis D. and Haugan M. (1997). Modeling the information seeking patterns of engineers and research scientists in an industrial environment. J. Document. 53: 384–403

    Article  Google Scholar 

  57. Foster A. and Ford N. (2003). Serendipity and information seeking: an empirical study. J. Document. 59: 321–340

    Article  Google Scholar 

  58. Belkin N.J., Oddy R.N. and Brooks H.M. (1982). ASK for information retrieval: part 1 Background and theory. J. Document. 38: 61–71

    Google Scholar 

  59. Wilson, T.D.: Exploring models of information behaviour: the uncertainty project. Inf. Process. Manage. 893–849 (1999)

  60. Wood R. (1996). The Conspectus: a collection analysis and development success. Libr. Acquis. Pract. Theory 20: 429–453

    Article  Google Scholar 

  61. Saracevic, T.: Evaluation of digital libraries: an overview. In: Agosti, M., Fuhr, N. (eds.) Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries, Padua, Italy (2004) http://dlib.ionio.gr/wp7/workshop2004_program.html

  62. Agosti, M., Ferro, N., Frommholz, I., Thiel, U.: Annotations in digital libraries and collaboratories: facets, models and usage. In: Lecture Notes In Computer Science, vol. 3232, pp. 244—255 Springer, Berlin (2004) Research and Advanced Technology for Digital Libraries: 8th European Conference, ECDL 2004. Proceedings

  63. Agosti, M., Albrechtsen, H., Ferro, N., Frommholz, I., Hansen, P., Orio, N., Panizzi, E., Pejtersen, A.M., Thiel, U.: Dilas: a digital library annotation service. In: International Workshop on Annotation for Collaboration. Methods, Tools, and Practices, pp. 91–101 (2005)

  64. Hansen, P., Jarvelin, K.: The information seeking and retrieval process at the swedish patent and registration office: moving from lab-based to real life work task environment. In: Agosti, M., Fuhr, N. (eds.) Proceedings of the ACM-SIGIR 2000 Workshop on Patent Retrieval, Athens, Greece (2000) 43–53 http://www.sics.se/∼preben/papers/ SIGIR20000-WS.pdf

  65. Covey, D.: Usage and usability assessment: library practices and concerns, vol. 105. CLIR (2002) http://www.clir.org/pubs/reports/pub105/pub105.pdf

  66. Borgman, C.L.: Social Aspects of Digital Libraries-Final Report to the NSF. Technical report (1996) http://is.gseis.ucla.edu/research/dl/UCLA_DL_Report.html

  67. Kovács, L., Micsik, A.: The evaluation computer: a model for structuring evaluation activities. In: Agosti, M., Fuhr, N. (eds.) DELOS Workshop on the Evaluation of Digital Libraries, Padova, Italy (2004) http://dlib.ionio.gr/wp7/ workshop2004_program.html

  68. Banwell, L., Coulson, G.: Users and user study methodology: the JUBILEE project. Inf. Res. 9, 47–56 (2004) http:// informationr.net/ir/9-2/paper167.html

    Google Scholar 

  69. Jeng J. (2005). What is usability in the context of the digital library and how can it be measured? Inf. Technol. Libr. 24: 47–56

    Google Scholar 

  70. Xie H. (2006). Evaluation of digital libraries: criteria and problems from users perspectives. Libr. Inf. Res. 28: 433–452

    Article  Google Scholar 

  71. Klas, C., Albrechtsen, H.N.F., Hansen, P., Kapidakis, S., Kovacs, L., Kriewel, S., Micsik, A., Papatheodorou, C., Tsakonas, G., Jacob, E.: A logging scheme for comparative digital library evaluation. In: Lecture Notes In Computer Science, vol. 4172, pp. 267—278 Springer, Berlin (2006) Research and Advanced Technology for Digital Libraries: 10th European Conference, ECDL 2006, Alicante, Spain, September 17—22, 2006. Proceedings

Download references

Author information

Authors and Affiliations

  1. University of Duisburg-Essen, Duisburg, Germany

    Norbert Fuhr & Claus-Peter Klas

  2. Norwegian University of Science and Technology, Trondheim, Norway

    Trond Aalberg & Ingeborg Sølvberg

  3. Swedish Institute of Computer Science, Kista, Sweden

    Preben Hansen

  4. Computer and Automation Research Institute of the Hungarian Academy of Sciences (MTA SZTAKI), Budapest, Hungary

    László Kovács & András Micsik

  5. University of Padua, Padova, Italy

    Maristella Agosti

  6. Department of Archive and Library Sciences, Ionian University, Corfu, Greece

    Giannis Tsakonas, Sarantos Kapidakis, Christos Papatheodorou & Ingeborg Sølvberg

  7. CNR-ISTI, Pisa, Italy

    Carol Peters

  8. Department of Computer and Information Sciences, University of Strathclyde, Glasgow, UK

    Monica Landoni

Authors
  1. Norbert Fuhr
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giannis Tsakonas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Trond Aalberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maristella Agosti
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Preben Hansen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sarantos Kapidakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Claus-Peter Klas
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. László Kovács
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Monica Landoni
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. András Micsik
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Christos Papatheodorou
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Carol Peters
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Ingeborg Sølvberg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Claus-Peter Klas.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fuhr, N., Tsakonas, G., Aalberg, T. et al. Evaluation of digital libraries. Int J Digit Libr 8, 21–38 (2007). https://doi.org/10.1007/s00799-007-0011-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00799-007-0011-z

Keywords

Search

Navigation