Skip to main content
SpringerLink
Log in

A Decade of Evaluating Europeana - Constructs, Contexts, Methods & Criteria

  • Conference paper
  • First Online:
Book cover Research and Advanced Technology for Digital Libraries (TPDL 2017)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10450))

Included in the following conference series:

Abstract

This meta-analysis of 41 evaluation studies of the Europeana Digital Library categorizes them by their constructs, contexts, criteria, and methodologies using Saracevic’s digital library evaluation framework. The analysis shows that system-centered evaluations prevail over user-centered evaluations and evaluations from a societal or institutional perspective are missing. The study reveals, which Europeana components have received focused attention in the last decade (e.g. the metadata) and can serve as a reference for identifying gaps, selecting methodologies and re-using data for future evaluations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Already twenty years ago, the first European Conference on Research and Advanced Technology for Digital Libraries (ECDL, now TPDL) was held in Pisa, Italy [39].

  2. 2.

    http://www.europeana.eu.

  3. 3.

    http://strategy2020.europeana.eu/.

  4. 4.

    Search terms: Europeana and (user* or evaluat* or study*) and variations.

  5. 5.

    http://pro.europeana.eu/get-involved/projects/project-list.

  6. 6.

    The study defines this term as a criteria to act as multiplier for member institutions.

  7. 7.

    More information can be found here: https://impkt.tools/.

  8. 8.

    http://statistics.europeana.eu/.

  9. 9.

    http://pro.europeana.eu/page/data-quality-committee.

References

  1. Akasereh, M., Malak, P., Pawłowski, A.: Evaluation of IR strategies for Polish. In: Przepiórkowski, A., Ogrodniczuk, M. (eds.) NLP 2014. LNCS, vol. 8686, pp. 384–391. Springer, Cham (2014). doi:10.1007/978-3-319-10888-9_38

    Google Scholar 

  2. van den Akker, C., van Nuland, A., van der Meij, L., van Erp, M., Legêne, S., Aroyo, L., Schreiber, G.: From information delivery to interpretation support: evaluating cultural heritage access on the web. In: WebSci 2013, pp. 431–440. ACM (2013)

    Google Scholar 

  3. Aletras, N., Stevenson, M.: Computing similarity between cultural heritage items using multimodal features. In: LaTeCH 2012, pp. 85–93. ACL (2012)

    Google Scholar 

  4. Aletras, N., Stevenson, M., Clough, P.: Computing similarity between items in a digital library of cultural heritage. J. Comp. Cult. Heri. 5(4), 16:1–16:19 (2012)

    Google Scholar 

  5. Aloia, N., Concordia, C., Gerwen, A.M., Hansen, P., Kuwahara, M., Ly, A.T., Meghini, C., Spyratos, N., Sugibuchi, T., Tanaka, Y., Yang, J., Zeni, N.: Design, implementation and evaluation of a user generated content service for Europeana. In: Gradmann, S., Borri, F., Meghini, C., Schuldt, H. (eds.) TPDL 2011. LNCS, vol. 6966, pp. 477–482. Springer, Heidelberg (2011). doi:10.1007/978-3-642-24469-8_54

    Chapter  Google Scholar 

  6. Candela, L., Castelli, D., Ferro, et al.: The DELOS digital library reference model. Version 0.96. Technical report, DELOS Network of Excellence on Digital Libraries (2007)

    Google Scholar 

  7. Ceccarelli, D., Gordea, S., Lucchese, C., Nardini, F.M., Tolomei, G.: Improving Europeana search experience using query logs. In: Gradmann, S., Borri, F., Meghini, C., Schuldt, H. (eds.) TPDL 2011. LNCS, vol. 6966, pp. 384–395. Springer, Heidelberg (2011). doi:10.1007/978-3-642-24469-8_39

    Chapter  Google Scholar 

  8. CIBER Research Ltd.: Europeana 2012–2013: usage and performance update. Technical report, CIBER Research Ltd. (2013). http://bit.ly/2rgWjrl

  9. Clark, D., Nicholas, I., Rowlands, I.: D3.1.3 Report on best-practice and how users are using the Europeana service. Technical report, D 3.1.3, EuropeanaConnect (2011)

    Google Scholar 

  10. Clough, P., Otegi, A., Agirre, E., Hall, M.: Implementing recommendations in the PATHS system. In: Bolikowski, Ł., Casarosa, V., Goodale, P., Houssos, N., Manghi, P., Schirrwagen, J. (eds.) TPDL 2013. CCIS, vol. 416, pp. 169–173. Springer, Cham (2014). doi:10.1007/978-3-319-08425-1_17

    Google Scholar 

  11. Şencan, I.: The performance evaluation of the information retrieval system of the Europeana website. In: IMCW 2013, p. 59 (2013)

    Google Scholar 

  12. Dangerfield, M.C., Kalshoven, L.: Report and recommendations from the task force on metadata quality. Technical report, Europeana (2015). http://bit.ly/2rM113M

  13. Dani, A., Chatzopoulou, C., Siatri, R., Mystakopoulos, F., Antonopoulou, S., Katrinaki, E., Garoufallou, E.: Digital libraries evaluation: measuring Europeana’s usability. In: Garoufallou, E., Hartley, R.J., Gaitanou, P. (eds.) MTSR 2015. CCIS, vol. 544, pp. 225–236. Springer, Cham (2015). doi:10.1007/978-3-319-24129-6_20

    Chapter  Google Scholar 

  14. Dickel, J.: Digitale Bibliotheken im Vergleich: Europeana & WDL. Perspektive Bibliothek 4(1), 45–67 (2015)

    Google Scholar 

  15. Dobreva, M., Chowdhury, S.: A user-centric evaluation of the Europeana digital library. In: Chowdhury, G., Koo, C., Hunter, J. (eds.) ICADL 2010. LNCS, vol. 6102, pp. 148–157. Springer, Heidelberg (2010). doi:10.1007/978-3-642-13654-2_19

    Chapter  Google Scholar 

  16. Dobreva, M., McCulloch, E., Birrell, D., Ünal, Y., Feliciati, P.: Digital natives and specialised digital libraries: a study of Europeana users. In: Kurbanoğlu, S., Al, U., Lepon Erdoğan, P., Tonta, Y., Uçak, N. (eds.) IMCW 2010. CCIS, vol. 96, pp. 45–60. Springer, Heidelberg (2010). doi:10.1007/978-3-642-16032-5_5

    Chapter  Google Scholar 

  17. Fox, E.A., Gonçalves, M.A., Shen, R.: Theoretical foundations for digital libraries: the 5s (societies, scenarios, spaces, structures, streams) approach. Synth. Lect. Inform. Concepts, Retrieval, Serv. 4(2), 1–180 (2012)

    Article  MATH  Google Scholar 

  18. Fuhr, N., Hansen, P., Mabe, M., Micsik, A., Sølvberg, I.: Digital libraries: a generic classification and evaluation scheme. In: Constantopoulos, P., Sølvberg, I.T. (eds.) ECDL 2001. LNCS, vol. 2163, pp. 187–199. Springer, Heidelberg (2001). doi:10.1007/3-540-44796-2_17

    Chapter  Google Scholar 

  19. Fuhr, N., Tsakonas, G., Aalberg, T., et al.: Evaluation of digital libraries. IJDL 8(1), 21–38 (2007)

    Google Scholar 

  20. Gäde, M.: Country and language level differences in multilingual digital libraries. Dissertation, Humboldt-Universität zu Berlin, April 2014

    Google Scholar 

  21. Gäde, M., Petras, V.: Multilingual interface preferences. In: IIiX 2014, pp. 231–234. ACM (2014)

    Google Scholar 

  22. Glaser, B.G., Strauss, A.: The discovery of ground theory (1967)

    Google Scholar 

  23. Gonçalves, M.A., Fox, E.A., Watson, L.T., Kipp, N.A.: Streams, structures, spaces, scenarios, societies (5s): a formal model for digital libraries. ACM TOIS 22(2), 270–312 (2004)

    Article  Google Scholar 

  24. Gonçalves, M.A., Moreira, B.L., Fox, E.A., Watson, L.T.: “What is a good digital library?” - a quality model for digital libraries. IPM 43(5), 1416–1437 (2007)

    Google Scholar 

  25. Gonzalez-Agirre, A., Rigau, G., Agirre, E., Aletras, N., Stevenson, M.: Why are these similar? Investigating item similarity types in a large digital library. JASIST 67(7), 1624–1638 (2016)

    Google Scholar 

  26. Griffiths, J., et al.: D5.2 evaluation of the second PATHS prototype. Technical report (2014). http://bit.ly/2rh3hwJ

  27. Hill, T., Charles, V., Stiller, J., Isaac, A.: Searching for inspiration: user needs and search architecture in Europeana collections. In: ASIS&T, pp. 1–7. Wiley (2016)

    Google Scholar 

  28. Isaac, A., Manguinhas, H., Stiller, J., Charles, V.: Final report on evaluation and enrichment. Technical report, Europeana (2015). http://bit.ly/2rcKuHr

  29. Joo, S., Xie, I.: Evaluation constructs and criteria for digital libraries: a document analysis. In: Recent Developments in the Design, Construction, and Evaluation of Digital Libraries: Case Studies, pp. 126–140. IGI Global (2013)

    Google Scholar 

  30. Kapidakis, S.: Comparing metadata quality in the Europeana context. In: PETRA 2012, p. 25:1–25:8. ACM (2012)

    Google Scholar 

  31. Király, P.: A metadata quality assurance framework. Technical report, Europeana (2015). https://pkiraly.github.io/metadata-quality-project-plan.pdf

  32. Manguinhas, H., Freire, N., Isaac, A., Stiller, J., Charles, V., Soroa, A., Simon, R., Alexiev, V.: Exploring comparative evaluation of semantic enrichment tools for cultural heritage metadata. In: Fuhr, N., Kovács, L., Risse, T., Nejdl, W. (eds.) TPDL 2016. LNCS, vol. 9819, pp. 266–278. Springer, Cham (2016). doi:10.1007/978-3-319-43997-6_21

    Chapter  Google Scholar 

  33. Marchionini, G.: Evaluating digital libraries: a longitudinal and multifaceted view. Libr. Trends 49(2), 304 (2000)

    Google Scholar 

  34. Moreira, B.L., Gonçalves, M.A., Laender, A.H., Fox, E.A.: Automatic evaluation of digital libraries with 5squal. J. Informetrics 3(2), 102–123 (2009)

    Article  Google Scholar 

  35. Navarrete, T.: Europeana as online cultural information service. Study Report (2016). http://bit.ly/2rcMsaT

  36. Nicholas, D., Clark, D.: The second digital transition: to the mobile space - an analysis of Europeana. Learn. Publishing 26(4), 240–252 (2013)

    Article  Google Scholar 

  37. Nicholas, D., Clark, D., Rowlands, I., Jamali, H.R.: Information on the Go: a case study of Europeana mobile users. JASIST 64(7), 1311–1322 (2013)

    Article  Google Scholar 

  38. Olensky, M., Stiller, J., Dröge, E.: Poisonous India or the importance of a semantic and multilingual enrichment strategy. In: Dodero, J.M., Palomo-Duarte, M., Karampiperis, P. (eds.) MTSR 2012. CCIS, vol. 343, pp. 252–263. Springer, Heidelberg (2012). doi:10.1007/978-3-642-35233-1_25

    Chapter  Google Scholar 

  39. Peters, C., Thanos, C. (eds.): ECDL 1997. LNCS, vol. 1324. Springer, Heidelberg (1997)

    Google Scholar 

  40. Petras, V., Bogers, T., Toms, E., Hall, M., Savoy, J., Malak, P., Pawłowski, A., Ferro, N., Masiero, I.: Cultural heritage in CLEF (CHiC) 2013. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 192–211. Springer, Heidelberg (2013). doi:10.1007/978-3-642-40802-1_23

    Chapter  Google Scholar 

  41. Petras, V., Ferro, N., Gäde, M., Isaac, A., Kleineberg, M., Masiero, I., Nicchio, M., Stiller, J.: Cultural Heritage in CLEF (CHiC) Overview 2012. In: CLEF (2012)

    Google Scholar 

  42. Petras, V., Hill, T., Stiller, J., Gäde, M.: Europeana - a search engine for digitised cultural heritage material. Datenbank-Spektrum 17(1), 41–46 (2017)

    Article  Google Scholar 

  43. Purday, J.: Think culture: Europeana.eu from concept to construction. Electron. Libr. 27(6), 919–937 (2009)

    Article  Google Scholar 

  44. Saracevic, T.: Digital library evaluation: toward an evolution of concepts. Libr. Trends 49(2), 350–369 (2000)

    Google Scholar 

  45. Saracevic, T.: Evaluation of digital libraries: an overview. In: Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries, Padua, Italy (2004)

    Google Scholar 

  46. Schindler, A., Gordea, S., Biessum, H.: The Europeana sounds music information retrieval pilot. In: Ioannides, M., Fink, E., Moropoulou, A., Hagedorn-Saupe, M., Fresa, A., Liestøl, G., Rajcic, V., Grussenmeyer, P. (eds.) EuroMed 2016. LNCS, vol. 10059, pp. 109–117. Springer, Cham (2016). doi:10.1007/978-3-319-48974-2_13

    Chapter  Google Scholar 

  47. Schlötterer, J., Seifert, C., Wagner, L., Granitzer, M.: A game with a purpose to access europe’s cultural treasure. ECIR 2015, 13–18 (2015)

    Google Scholar 

  48. Schweibenz, W.: Eine erste evaluation der Europeana: Wie Benutzer das look & feel des prototypen der Europäischen digitalen bibliothek beurteilen. Inf. Wissenschaft und Praxis 61(5), 277–284 (2010)

    Google Scholar 

  49. Stiller, J., Gäde, M., Petras, V.: Multilingual access to digital libraries: the Europeana use case. Inf.-Wissenschaft und Praxis 64, 86–95 (2013)

    Google Scholar 

  50. Stiller, J.: From curation to collaboration. Dissertation, Humboldt-Universität zu Berlin, Philosophische Fakultät I (2014)

    Google Scholar 

  51. Stiller, J., Isaac, A., Petras, V.: EuropeanaTech task force on a multilingual and semantic enrichment strategy. Technical report, Europeana (2014). http://bit.ly/2soSHs7

  52. Stiller, J., Olensky, M., Petras, V.: A framework for the evaluation of automatic metadata enrichments. In: Closs, S., Studer, R., Garoufallou, E., Sicilia, M.-A. (eds.) MTSR 2014. CCIS, vol. 478, pp. 238–249. Springer, Cham (2014). doi:10.1007/978-3-319-13674-5_23

    Google Scholar 

  53. Stiller, J., Petras, V.: A framework for classifying and comparing interactions in cultural heritage information systems. In: Cultural Heritage Information: Access and management. Facet Publishing (2015)

    Google Scholar 

  54. Stiller, J., Petras, V., Gäde, M., Isaac, A.: Automatic enrichments with controlled vocabularies in Europeana: challenges and consequences. In: Ioannides, M., Magnenat-Thalmann, N., Fink, E., Žarnić, R., Yen, A.-Y., Quak, E. (eds.) EuroMed 2014. LNCS, vol. 8740, pp. 238–247. Springer, Cham (2014). doi:10.1007/978-3-319-13695-0_23

    Google Scholar 

  55. Sykes, J., Dobreva, M., Birrell, D., McCulloch, E., Ruthven, I., Ünal, Y., Feliciati, P.: A new focus on end users: eye-tracking analysis for digital libraries. In: Lalmas, M., Jose, J., Rauber, A., Sebastiani, F., Frommholz, I. (eds.) ECDL 2010. LNCS, vol. 6273, pp. 510–513. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15464-5_69

    Chapter  Google Scholar 

  56. Tsakonas, G., Mitrelis, A., Papachristopoulos, L., Papatheodorou, C.: An exploration of the digital library evaluation literature based on an ontological representation. JASIST 64(9), 1914–1926 (2013)

    Article  Google Scholar 

  57. Tsakonas, G., Papatheodorou, C.: An ontological representation of the digital library evaluation domain. JASIST 62(8), 1577–1593 (2011)

    Article  Google Scholar 

  58. Valtysson, B.: Europeana - the digital construction of Europe’s collective memory. Inf. Commun. Soc. 15(2), 151–170 (2012)

    Article  Google Scholar 

  59. Xie, H.: Evaluation of digital libraries: criteria and problems from users’ perspectives. Libr. Inf. Sci. Res. 28(3), 433–452 (2006)

    Article  Google Scholar 

  60. Xie, H.I.: Users’ evaluation of digital libraries (DLs): their uses, their criteria, and their assessment. IPM 44(3), 1346–1373 (2008)

    Google Scholar 

  61. Xie, I., Matusiak, K.K.: Chapter 10 - evaluation of digital libraries. In: Discover Digital Libraries, pp. 281–318. Elsevier, Oxford (2016)

    Google Scholar 

  62. Yankova, I., Velev, E., Nusheva, K., Sonia, S.: The European digital library-factor for long-life learning in arts and cultural studies. Qual. Quant. Methods Libr. 4(4), 965–971 (2015)

    Google Scholar 

  63. Zhang, Y.: Developing a holistic model for digital library evaluation. JASIST 61(1), 88–110 (2010)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Berlin School of Library and Information Science, Humboldt-Universität zu Berlin, Dorotheenstr. 26, 10117, Berlin, Germany

    Vivien Petras & Juliane Stiller

Authors
  1. Vivien Petras
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juliane Stiller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vivien Petras .

Editor information

Editors and Affiliations

  1. Faculteit der Geesteswetenschappen, Universiteit van Amsterdam , Amsterdam, The Netherlands

    Jaap Kamps

  2. Library & Information Center, University of Patras , Patras, Greece

    Giannis Tsakonas

  3. Aristotle University of Thessaloniki , Thessaloniki, Greece

    Yannis Manolopoulos

  4. Civil Engineering, University of Thrace , Kimmeria, Greece

    Lazaros Iliadis

  5. Informatics, Ionian University , Kerkyra, Greece

    Ioannis Karydis

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Petras, V., Stiller, J. (2017). A Decade of Evaluating Europeana - Constructs, Contexts, Methods & Criteria. In: Kamps, J., Tsakonas, G., Manolopoulos, Y., Iliadis, L., Karydis, I. (eds) Research and Advanced Technology for Digital Libraries. TPDL 2017. Lecture Notes in Computer Science(), vol 10450. Springer, Cham. https://doi.org/10.1007/978-3-319-67008-9_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-67008-9_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-67007-2

  • Online ISBN: 978-3-319-67008-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation