Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Living Lab Evaluation for Life and Social Sciences Search Platforms - LiLAS at CLEF 2021

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12657))

Included in the following conference series:

Abstract

Meta-evaluation studies of system performances in controlled offline evaluation campaigns, like TREC and CLEF, show a need for innovation in evaluating IR-systems. The field of academic search is no exception to this. This might be related to the fact that relevance in academic search is multi-layered and therefore the aspect of user-centric evaluation is becoming more and more important. The Living Labs for Academic Search (LiLAS) lab aims to strengthen the concept of user-centric living labs for the domain of academic search by allowing participants to evaluate their retrieval approaches in two real-world academic search systems from the life sciences and the social sciences. To this end, we provide participants with metadata on the systems’ content as well as candidate lists with the task to rank the most relevant candidate to the top. Using the STELLA-infrastructure, we allow participants to easily integrate their approaches into the real-world systems and provide the possibility to compare different approaches at the same time.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://ir.nist.gov/covidSubmit/archive.html.

  2. 2.

    https://livivo.de.

  3. 3.

    https://zbmed.de.

  4. 4.

    https://datasetsearch.research.google.com/.

  5. 5.

    https://search.gesis.org/.

References

  1. Armstrong, T.G., Moffat, A., Webber, W., Zobel, J.: Improvements that don’t add up: ad-hoc retrieval results since 1998. In: Proceeding of the 18th ACM Conference on Information and Knowledge Management, CIKM 2009, pp. 601–610. ACM, Hong Kong (2009). https://doi.org/10.1145/1645953.1646031

  2. Balog, K., Schuth, A., Dekker, P., Schaer, P., Tavakolpoursaleh, N., Chuang, P.Y.: Overview of the trec 2016 open search track. In: Proceedings of the Twenty-Fifth Text REtrieval Conference (TREC 2016). NIST (2016)

    Google Scholar 

  3. Breuer, T., Schaer, P., Tavakolpoursaleh, N., Schaible, J., Wolff, B., Müller, B.: STELLA: towards a framework for the reproducibility of online search experiments. In: Clancy, R., Ferro, N., Hauff, C., Lin, J., Sakai, T., Wu, Z.Z. (eds.) Proceedings of the Open-Source IR Replicability Challenge Co-Located with 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, OSIRRC@SIGIR 2019, Paris, France, July 25, 2019. CEUR Workshop Proceedings, vol. 2409, pp. 8–11. CEUR-WS.org (2019). http://ceur-ws.org/Vol-2409/position01.pdf

  4. Carevic, Z., Schaer, P.: On the connection between citation-based and topical relevance ranking: results of a pretest using iSearch. In: Proceedings of the First Workshop on Bibliometric-Enhanced Information Retrieval Co-Located with 36th European Conference on Information Retrieval (ECIR 2014), Amsterdam, The Netherlands, April 13, 2014. CEUR Workshop Proceedings, vol. 1143, pp. 37–44. CEUR-WS.org (2014). http://ceur-ws.org/Vol-1143/paper5.pdf

  5. Fuhr, N.: Some common mistakes in IR evaluation, and how they can be avoided. SIGIR Forum 51(3), 32–41 (2018). https://doi.org/10.1145/3190580.3190586

    Article  Google Scholar 

  6. Hienert, D., Kern, D., Boland, K., Zapilko, B., Mutschke, P.: A digital library for research data and related information in the social sciences. In: 19th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2019, Champaign, IL, USA, June 2–6, 2019, pp. 148–157. IEEE (2019). https://doi.org/10.1109/JCDL.2019.00030

  7. Hopfgartner, F., et al.: Continuous evaluation of large-scale information access systems: a case for living labs. Information Retrieval Evaluation in a Changing World. TIRS, vol. 41, pp. 511–543. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22948-1_21

    Chapter  Google Scholar 

  8. Jagerman, R., Balog, K., de Rijke, M.: Opensearch: lessons learned from an online evaluation campaign. J. Data Inf. Qual. 10(3), 13:1–13:15 (2018). https://doi.org/10.1145/3239575

  9. Kramer, A.D.I., Guillory, J.E., Hancock, J.T.: Experimental evidence of massive-scale emotional contagion through social networks. Proc. Natl. Acad. Sci. 111(24), 8788–8790 (2014). https://doi.org/10.1073/pnas.1320040111

    Article  Google Scholar 

  10. Lommatzsch, A., Kille, B., Hopfgartner, F., Ramming, L.: Newsreel multimedia at mediaeval 2018: news recommendation with image and text content. In: Working Notes Proceedings of the MediaEval 2018 Workshop. CEUR-WS (2018)

    Google Scholar 

  11. Radlinski, F., Kurup, M., Joachims, T.: How does clickthrough data reflect retrieval quality? In: Proceeding of the 17th ACM Conference on Information and Knowledge Mining - CIKM 2008, p. 43. ACM Press, Napa Valley (2008). https://doi.org/10.1145/1458082.1458092

  12. Schaer, P., Schaible, J., Garcia Castro, L.J.: Overview of LiLAS 2020 – living labs for academic search. In: Arampatzis, A., et al. (eds.) CLEF 2020. LNCS, vol. 12260, pp. 364–371. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58219-7_24

    Chapter  Google Scholar 

  13. Schaer, P., Schaible, J., Müller, B.: Living labs for academic search at CLEF 2020. In: Jose, J.M., et al. (eds.) ECIR 2020. LNCS, vol. 12036, pp. 580–586. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45442-5_75

    Chapter  Google Scholar 

  14. Schuth, A., Balog, K., Kelly, L.: Extended overview of the living labs for information retrieval evaluation (LL4IR) CLEF lab 2015. In: Cappellato, L., Ferro, N., Jones, G.J.F., SanJuan, E. (eds.) Working Notes of CLEF 2015 - Conference and Labs of the Evaluation forum, Toulouse, France, September 8–11, 2015. CEUR Workshop Proceedings, vol. 1391. CEUR-WS.org (2015). http://ceur-ws.org/Vol-1391/inv-pap8-CR.pdf

  15. de Solla Price, D.J.: Little Science, Big Science. Columbia University Press, New York (1963)

    Book  Google Scholar 

  16. Voorhees, E.M., et al.: TREC-COVID: constructing a pandemic information retrieval test collection. CoRR abs/2005.04474 (2020). https://arxiv.org/abs/2005.04474

  17. Yang, W., Lu, K., Yang, P., Lin, J.: Critically examining the “neural hype”: weak baselines and the additivity of effectiveness gains from neural ranking models. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR 2019, pp. 1129–1132. ACM Press, Paris (2019). https://doi.org/10.1145/3331184.3331340

Download references

Author information

Authors and Affiliations

  1. TH Köln - University of Applied Sciences, Cologne, Germany

    Philipp Schaer

  2. GESIS - Leibniz Institute for the Social Sciences, Cologne, Germany

    Johann Schaible

  3. ZB MED - Information Centre for Life Sciences, Cologne, Germany

    Leyla Jael Castro

Authors
  1. Philipp Schaer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Johann Schaible
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leyla Jael Castro
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Philipp Schaer .

Editor information

Editors and Affiliations

  1. Radboud University Nijmegen, Nijmegen, The Netherlands

    Djoerd Hiemstra

  2. Department of Computer Science, Katholieke Universiteit Leuven, Heverlee, Belgium

    Marie-Francine Moens

  3. Toulouse, Toulouse Institute of Computer Science Research, Toulouse, France

    Josiane Mothe

  4. Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche, Pisa, Italy

    Raffaele Perego

  5. Leipzig University, Leipzig, Germany

    Martin Potthast

  6. Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche, Pisa, Italy

    Fabrizio Sebastiani

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Schaer, P., Schaible, J., Castro, L.J. (2021). Living Lab Evaluation for Life and Social Sciences Search Platforms - LiLAS at CLEF 2021. In: Hiemstra, D., Moens, MF., Mothe, J., Perego, R., Potthast, M., Sebastiani, F. (eds) Advances in Information Retrieval. ECIR 2021. Lecture Notes in Computer Science(), vol 12657. Springer, Cham. https://doi.org/10.1007/978-3-030-72240-1_77

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-72240-1_77

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-72239-5

  • Online ISBN: 978-3-030-72240-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation