skip to main content
research-article

Crowdsourcing for relevance evaluation

Published: 30 November 2008 Publication History

Abstract

Relevance evaluation is an essential part of the development and maintenance of information retrieval systems. Yet traditional evaluation approaches have several limitations; in particular, conducting new editorial evaluations of a search system can be very expensive. We describe a new approach to evaluation called TERC, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.

References

[1]
Amazon Mechanical Turk, http://www.mturk.com
[2]
Jeff Barr and Luis Felipe Cabrera. "AI Gets a Brain", ACM Queue, May 2006.
[3]
Brendan O'Connor, "Search Engine Relevance: An Empirical Test", http://blog.doloreslabs.com/2008/04/search-engine-relevance-an-empirical-test/#more-35, accessed April 13, 2008.
[4]
Jeff Howe. "The Rise of Crowdsourcing". Wired, June 2006. http://www.wired.com/wired/archive/14.06/crowds.html
[5]
Peter Ingwersen and Kalervo Järvelin. The Turn: Integration of Information Seeking and Retrieval in Context, Springer, 2005.
[6]
Thorsten Joachims and Filip Radlinski, "Search Engines that Learn from Implicit Feedback", IEEE Computer, Vol. 40, No. 8, August 2007.
[7]
Daniel E. Rose, "Why Is Web Search So Hard. to Evaluate?" Journal of Web Engineering, Vol. 3, Nos. 3 & 4, pp. 171--181, December 2004.
[8]
Tefko Saracevic. "Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part III: Behavior and Effects on Relevance". Journal of the American Society for Information Science and Technology, 58(13):212--2144, 2007.
[9]
Ellen Voorhees. "TREC: Continuing Information Retrieval's Tradition of Experimentation". Comm. Of the ACM, Vol. 50, No. 11, November 2007.

Cited By

View all
  • (2024)Crowdsourcing: An Education FrameworkInternational Journal of Innovative Science and Research Technology (IJISRT)10.38124/ijisrt/IJISRT24MAR1881(2229-2234)Online publication date: 8-Apr-2024
  • (2024)Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657712(1952-1962)Online publication date: 10-Jul-2024
  • (2024)When Search Engine Services meet Large Language Models: Visions and ChallengesIEEE Transactions on Services Computing10.1109/TSC.2024.3451185(1-23)Online publication date: 2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM SIGIR Forum
ACM SIGIR Forum  Volume 42, Issue 2
December 2008
101 pages
ISSN:0163-5840
DOI:10.1145/1480506
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 November 2008
Published in SIGIR Volume 42, Issue 2

Check for updates

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)28
  • Downloads (Last 6 weeks)2
Reflects downloads up to 16 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Crowdsourcing: An Education FrameworkInternational Journal of Innovative Science and Research Technology (IJISRT)10.38124/ijisrt/IJISRT24MAR1881(2229-2234)Online publication date: 8-Apr-2024
  • (2024)Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657712(1952-1962)Online publication date: 10-Jul-2024
  • (2024)When Search Engine Services meet Large Language Models: Visions and ChallengesIEEE Transactions on Services Computing10.1109/TSC.2024.3451185(1-23)Online publication date: 2024
  • (2024)Uncovering labeler bias in machine learning annotation tasksAI and Ethics10.1007/s43681-024-00572-wOnline publication date: 16-Sep-2024
  • (2023)Preference-Based Offline EvaluationProceedings of the Sixteenth ACM International Conference on Web Search and Data Mining10.1145/3539597.3572725(1248-1251)Online publication date: 27-Feb-2023
  • (2023)Automatic Linking of Podcast Segments to Topically Related WebpagesArtificial Intelligence and Cognitive Science10.1007/978-3-031-26438-2_30(381-393)Online publication date: 23-Feb-2023
  • (2022)In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd WorkersFrontiers in Artificial Intelligence10.3389/frai.2022.8281875Online publication date: 18-May-2022
  • (2022)CrowdDeep: deep-learning from the crowd for nuclei segmentationMedical Imaging 2022: Digital and Computational Pathology10.1117/12.2622862(56)Online publication date: 4-Apr-2022
  • (2022)Fair compensation of crowdsourcing work: the problem of flat ratesBehaviour & Information Technology10.1080/0144929X.2022.215056442:16(2871-2892)Online publication date: 28-Nov-2022
  • (2022)A conceptual framework for crowdsourcing requirements engineering in SCRUM‐based environmentIET Software10.1049/sfw2.12077Online publication date: 22-Nov-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media