Evaluation of Digital Libraries: An Insight into Useful Applications and Methods

We live in an age of evaluation; everything is measured, or an attempt is made to do so. This has its risks because practical‐to‐collect metrics can be confused with the real qualitative value of a service to the individual, organisation or society. Developing a sophisticated understanding of evaluation is an important area of effort, therefore. The book reviewed makes a valuable contribution by collecting together a wide range of material representing current thinking on digital library evaluation. The book certainly assembles work by some of the major figures working in the field, such as Saracevic, who is widely cited as the first to ask questions about the nature of digital library evaluation, Kyrillidou and colleagues who are leading The Association of Research Libraries activity in developing e‐metrics and Nicholas with his concept of deep‐log analysis.

The book is organised in four parts, intended, according to the editors, to address the: who, what, why and how of evaluation. This is only a loose framework, e.g. I would have thought that deep‐log analysis is chiefly a “how” of evaluation, but the chapter on it appears in the “what” section. In fact, many of the chapters address a number of the questions, so I was not sure how necessary or helpful the framework was.

After a short preface by the editors, which includes a summary of each chapter, Saracevic gives a useful introduction to evaluation as a whole.

Part one of the book has two chapters exploring the perspectives of some of those concerned with evaluation, namely, librarians and funders. The Franklin, Kyrillidou and Plum chapter gives a really good introduction to the range of library measurement initiatives in the electronic resource management field: COUNTER, MINES etc. I found the next chapter less useful, if only because it is a case study of an American funder, therefore it is less immediately relevant in a UK context. It was useful to explore the funder perspective, however. Obviously, the two chapters do not fully cover the question of “who” might have a concern with evaluation.

Part two looks at four objects of evaluation, namely usability, usefulness, performance and usage. Jeng offers a framework for understanding usability, while Garoufallou, Siatri and Hartley explore the literature on the use of digital libraries. Agosti and Ferro propose a process by which evaluation data is itself systematically collected for reuse. Nicholas describes and illustrates deep‐log analysis.

According to the editors, Part three addresses the purposes of evaluation. I was not entirely convinced by this characterisation, but there is some useful content here. Blandford and Bainbridge explore the differences between user studies led and systems led development. This articulates what seems to me to be a key tension in thinking about digital libraries. The authors give a good sense of the diverse approaches available in developing an understanding of what users do/need. In the next chapter, the editors of the book, Tsakonas and Papatheodorou explore outcomes assessment, which moves us beyond the more immediate evaluation criteria of use levels and usability, looking more broadly at the contribution of the digital library to the organisational mission. This is likely to be the direction digital library evaluation will increasingly take, within wider assessment cultures. The editors could have been forgiven for giving more prominence to their own chapter, I think. Kyrillidou, Cook and Lincoln summarise the major dimensions of evaluation for digital library service quality discovered in the focus groups held as part of the process of developing DigiQUAL.

According to the editors' outline, Part four looks at methods of evaluation. Khoo and Giersch propose ways to conduct evaluation as a process, e.g. in terms of setting criteria and obtaining buy‐in. Monopoli looks at qualitative methods and issues around choice among them. Theng gives some examples of analysis of quantitative questionnaire data.

Edited books often contain a rather incoherent collection of chapters, which essentially can only be read separately, because each author has a different understanding of the core concept and because there is no one to provide an overview or framework. In the case of this book, the concepts of digital libraries and of evaluation are rather complex and contested. For example, the book contains material both on digital libraries as discrete locally managed collections and also on the rather different challenges posed by managing access to extremely large collections of licensed resources. Thus in Part one, the first chapter was clearly in the electronic resource management area, whereas the other was more obviously tied to digital libraries. Having acknowledged this general issue, on the whole the book succeeds better than most edited collections in offering the reader access to key issues around the subject. A lot is still left to the reader to synthesise the different viewpoints, of course. The balance is probably rather more towards Saracevic's (2000) lower levels of evaluation, such as of interface, engineering, process and content. Social and institutional levels of evaluation are less well developed. Considering evaluation from a “balanced scorecard” perspective, service and process are addressed in the book, but there is little about financial evaluation or the “innovation and learning” quadrant. These gaps, however, seem to exist in current practices, not just in the book. Certainly the editors' intention to assemble authors who offer both academic rigor and accessibility is successful.