Skip to main content
Springer Nature Link
Log in

Result Assessment Tool: Software to Support Studies Based on Data from Search Engines

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14612))

Included in the following conference series:

Abstract

The Result Assessment Tool (RAT) is a software toolkit for conducting research with results from commercial search engines and other information retrieval (IR) systems. The software integrates modules for study design and management, automatic collection of search results via web scraping, and evaluation of search results in an assessment interface using different question types. RAT can be used for conducting a wide range of studies, including retrieval effectiveness studies, classification studies, and content analyses.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The SEO-classifier implementation is described at [16].

  2. 2.

    We developed a script that generates search queries based on keyword suggestions generated by the Google Ads API: https://developers.google.com/google-ads/api/.

  3. 3.

    Repository for the research data generated with RAT: https://osf.io/t3hg9/

  4. 4.

    The RAT software demo is available at https://rat-software.org/

  5. 5.

    Repository for the source code: https://github.com/rat-software/

References

  1. Bar-Ilan, J., Levene, M.: A method to assess search engine results. Online Inf. Rev. 35, 854–868 (2011). https://doi.org/10.1108/14684521111193166

    Article  Google Scholar 

  2. Tawileh, W., Griesbaum, J., Mandl, T.: Evaluation of five web search engines in Arabic language. In: Atzmüller, M., Benz, D., Hotho, A., and Stumme, G. (eds.) Proceedings of LWA 2010, Kassel, Germany, pp. 1–8 (2010)

    Google Scholar 

  3. Trielli, D., Diakopoulos, N.: Partisan search behavior and Google results in the 2018 U.S. midterm elections. Inf. Commun. Soc. 1–17 (2020). https://doi.org/10.1080/1369118X.2020.1764605

  4. Lingnau, A., Ruthven, I., Landoni, M., van der Sluis, F.: Interactive search interfaces for young children - the PuppyIR approach. In: 2010 10th IEEE International Conference on Advanced Learning Technologies, pp. 389–390. IEEE (2010). https://doi.org/10.1109/ICALT.2010.111

  5. Renaud, G., Azzopardi, L.: SCAMP. In: Proceedings of the 4th Information Interaction in Context Symposium on - IIIX 2012, pp. 286–289. ACM Press, New York (2012). https://doi.org/10.1145/2362724.2362776

  6. Dussin, M., Ferro, N.: Design of a digital library system for large-scale evaluation campaigns. In: Christensen-Dalsgaard, B., Castelli, D., Ammitzbøll Jurik, B., Lippincott, J. (eds.) Research and Advanced Technology for Digital Libraries, pp. 400–401. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-87599-4_45

    Chapter  Google Scholar 

  7. Koopman, B.: Semantic search as inference. ACM SIGIR Forum. (2014). https://doi.org/10.1145/2701583.2701601

    Article  Google Scholar 

  8. Ogilvie, P., Callan, J.P.: Experiments using the lemur toolkit. In: Proceedings of Tenth Text REtrieval Conference, TREC 2001, Gaithersburg, MD, USA, 13–16 November 2001 (2001)

    Google Scholar 

  9. Digitalmethods: DMI Tools. https://wiki.digitalmethods.net/Dmi/ToolDatabase. Accessed 23 Feb 2023

  10. Thelwall, M.: Introduction to webometrics: quantitative web research for the social sciences. Synth. Lect. Inf. Concepts Retr. Serv. (2009). https://doi.org/10.2200/s00176ed1v01y200903icr004

    Article  Google Scholar 

  11. Janssen, S., Käsmann, L., Fahlbusch, F.B., Rades, D., Vordermark, D.: Side effects of radiotherapy in breast cancer patients: the Internet as an information source. Strahlenther. Onkol. Organ Dtsch. Rontgengesellschaft Al 194, 136–142 (2018). https://doi.org/10.1007/s00066-017-1197-7

    Article  Google Scholar 

  12. Rachul, C., Marcon, A.R., Collins, B., Caulfield, T.: COVID-19 and ‘immune boosting’ on the internet: a content analysis of Google search results. BMJ Open 10, e040989 (2020). https://doi.org/10.1136/bmjopen-2020-040989

    Article  Google Scholar 

  13. Ballatore, A.: Google chemtrails: a methodology to analyze topic representation in search engine results. First Monday 20 (2015)

    Google Scholar 

  14. Lewandowski, D.: Evaluating the retrieval effectiveness of web search engines using a representative query sample. J. Assoc. Inf. Sci. Technol. 66, 1763–1775 (2015). https://doi.org/10.1002/asi.23304

    Article  Google Scholar 

  15. Hinz, K., Sünkler, S., Lewandowski, D.: SEO im Wahlkampf. In: Korte, K.-R., Schiffers, M., von Schuckmann, A., Plümer, S. (eds.) Die Bundestagswahl 2021, pp. 1–28. Springer, Wiesbaden (2023). https://doi.org/10.1007/978-3-658-35758-0_19-1

  16. Lewandowski, D., Sünkler, S., Yagci, N.: The influence of search engine optimization on Google’s results: a multi-dimensional approach for detecting SEO. In: 13th ACM Web Science Conference 2021, WebSci 2021, 21–25 June 2021 Virtual Event UK (2021). https://doi.org/10.1145/3447535.3462479

  17. Haider, J., Ekström, B., Wallin, E.T., Lorentzen, D.G., Rödl, M., Söderberg, N.: Tracing online information about wind power in Sweden: an exploratory quantitative study of broader trends (2023). https://doi.org/10.13140/RG.2.2.27914.13766

  18. Schultheiß, S., Lewandowski, D., Von Mach, S., Yagci, N.: Query sampler: generating query sets for analyzing search engines using keyword research tools. PeerJ Comput. Sci. 9, e1421 (2023). https://doi.org/10.7717/peerj-cs.1421

    Article  Google Scholar 

  19. Yagci, N., Sünkler, S., Häußler, H., Lewandowski, D.: A comparison of source distribution and result overlap in web search engines. Proc. Assoc. Inf. Sci. Technol. 59, 346–357 (2022). https://doi.org/10.1002/pra2.758

    Article  Google Scholar 

  20. Abras, C., Maloney-krichmar, D., Preece, J.: User-centered design. In: Bainbridge, W. (ed.) Encyclopedia of Human-Computer Interaction, pp. 445–456. Sage Publications, Thousand Oaks (2004)

    Google Scholar 

  21. International Organization for Standardization: ISO 9241-210 (2019). https://www.iso.org/standard/77520.html. Accessed 10 Oct 2023

  22. Wilkinson, M.D., et al.: The FAIR guiding principles for scientific data management and stewardship. Sci. Data. 3, 160018 (2016). https://doi.org/10.1038/sdata.2016.18

    Article  Google Scholar 

Download references

Acknowledgments

This work is funded by the German Research Foundation (Deutsche Forschungsgemeinschaft (DFG); Grant No. 460676551).

Author information

Authors and Affiliations

  1. Department Information, Media and Communication, Hamburg University of Applied Sciences, Finkenau 35, 22081, Hamburg, Germany

    Sebastian Sünkler, Nurce Yagci, Sebastian Schultheiß, Sonja von Mach & Dirk Lewandowski

  2. University of Duisburg-Essen, Campus Duisburg, 47057, Duisburg, Germany

    Dirk Lewandowski

Authors
  1. Sebastian Sünkler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nurce Yagci
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sebastian Schultheiß
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sonja von Mach
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dirk Lewandowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sebastian Sünkler .

Editor information

Editors and Affiliations

  1. Georgetown University, Washington, WA, USA

    Nazli Goharian

  2. University of Pisa, PISA, Pisa, Italy

    Nicola Tonellotto

  3. King's College London, London, UK

    Yulan He

  4. University College London, London, UK

    Aldo Lipani

  5. University of Glasgow, Glasgow, UK

    Graham McDonald

  6. University of Glasgow, Glasgow, UK

    Craig Macdonald

  7. University of Glasgow, Glasgow, UK

    Iadh Ounis

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sünkler, S., Yagci, N., Schultheiß, S., von Mach, S., Lewandowski, D. (2024). Result Assessment Tool: Software to Support Studies Based on Data from Search Engines. In: Goharian, N., et al. Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol 14612. Springer, Cham. https://doi.org/10.1007/978-3-031-56069-9_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56069-9_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56068-2

  • Online ISBN: 978-3-031-56069-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics