Users’ evaluation of digital libraries (DLs): Their uses, their criteria, and their assessment

Abstract

Millions of dollars have been invested into the development of digital libraries. There are many unanswered questions regarding their evaluation, in particular, from users’ perspectives. This study intends to investigate users’ use, their criteria and their evaluation of the two selected digital libraries. Nineteen subjects were recruited to participate in the study. They were instructed to keep a diary for their use of the two digital libraries, rate the importance of digital library evaluation criteria, and evaluate the two digital libraries by applying their perceived important criteria. The results show patterns of users’ use of digital libraries, their perceived important evaluation criteria, and the positive and negative aspects of digital libraries. Finally, the relationships between perceived importance of digital library evaluation criteria and actual evaluation of digital libraries and the relationships between use of digital libraries and evaluation of digital libraries as well as users’ preference, experience and knowledge structure on digital library evaluation are further discussed.

Introduction

The availability of the Internet brings dramatic changes to millions of people in terms of how they collect, organize, disseminate, access, and use information. Perceptions of digital libraries vary and evolve over time, and many definitions for digital libraries have been proposed. The concept of a digital library means different things to different people. Even the key players in the development and use of digital libraries have different understanding of digital libraries. To librarians, a digital library is another form of a physical library; to computer scientists, a digital library is a distributed text-based information system or a networked multimedia information system; to end users, digital libraries are similar to the world wide web (www) with improvements in performance, organization, functionality, and usability (Fox, Akscyn, Furuta, & Leggett, 1995). Borgman’s (1999) two competing visions of digital libraries stimulate more discussions on the definition of a digital library by researchers and practitioners. The common elements of a digital library definition identified by the Association of Research Libraries (1995) are more acceptable to researchers of digital libraries:

• The digital library is not a single entity.

• The digital library requires technology to link the resources of many.

• The linkages between the many digital libraries and information services are transparent to the end users.

• Universal access to digital libraries and information services is a goal.

• Digital library collections are not limited to document surrogates: they extend to digital artifacts that cannot be represented or distributed in printed formats.

The Digital Library Initiative 1 & II funded by National Science Foundation (NSF) and other federal agencies have advanced the technical as well as the social, behavioral, and economic research needed to design and develop digital libraries. Millions of dollars have been invested into the development of digital libraries. Many unanswered questions related to whether users use them, how they use them, and what facilitate and hinder their access of information in these digital libraries. These questions cannot be answered without the evaluation of the existing digital libraries. We need to assess the usability of digital libraries in order to evaluate the full potential of digital libraries (Blandford & Buchanan, 2003). Moreover, user model of digital libraries and digital library model of users are different (Saracevic, 2004). It is important to understand users’ perspectives of digital libraries. In addition, needs assessment and evaluation is also essential for the iterative design for digital libraries (Van House, Butler, Ogle, & Schiff, 1996). In order to answer these questions, we need to first clarify the following questions: (1) what are the digital library (DL) evaluation criteria? (2) who determines these criteria? or how these criteria are determined?

Just as definitions for digital libraries vary, so do the DL evaluation criteria. A major challenge for DL evaluation is to identify what to evaluate and how to evaluate non-intrusively at low cost (Borgamn & Larsen, 2003). DL evaluation is a challenging task due to the complicated technology, rich content and a variety of users involved (Borgman et al., 2000, Saracevic and Covi, 2000). The most recognized DL evaluation criteria are derived from evaluation criteria for traditional libraries, IR system performance, and human-computer interaction (Chowdhury and Chowdhury, 2003, Marchionini, 2000, Saracevic, 2000, Saracevic and Covi, 2000). Very few studies actually apply all the DL evaluation criteria to assess a digital library. Many of the studies focus on the evaluation of usability of digital libraries. After reviewing usability tests in selected academic digital libraries, Jeng, 2005a, Jeng, 2005b found that ease of use, satisfaction, efficiency and effectiveness are the main applied criteria. Some of the evaluation studies extend to assess performance, content and services of digital libraries while service evaluation mainly concentrates on digital reference (Carter & Janes, 2000). Other evaluation studies also look into the impact of digital libraries (Marchionini, 2000).

Little research has investigated users’ evaluation of digital libraries, in particular, their criteria and their actual assessment of digital libraries. Xie (2006) pointed out that even though researcher have developed DL evaluation criteria, and conducted actual studies to evaluate existing digital libraries or prototypes of digital libraries, there is a lack of user input regarding evaluation criteria. User evaluation of digital libraries by applying their own criteria is needed. The remaining questions related to DL evaluation are: What is the relationship between users’ use and evaluation of digital libraries? And what is the relationship between users’ perceived DL evaluation criteria and their actual evaluation?