skip to main content
10.1145/2499178.2499198acmotherconferencesArticle/Chapter ViewAbstractPublication PagesictirConference Proceedingsconference-collections
poster

A Modification of LambdaMART to Handle Noisy Crowdsourced Assessments

Published: 29 September 2013 Publication History

Abstract

We consider noisy crowdsourced assessments and their impact on learning-to-rank algorithms. Starting with EM-weighted assessments, we modify LambdaMART in order to use smoothed probabilistic preferences over pairs of documents, directly as input to the ranking algorithm.

References

[1]
C.J.C. Burges. From ranknet to lambdarank to lambdamart: An overview, 2010.
[2]
O. Chapelle and Y. Chang. Yahoo! learning to rank challenge overview. JMLR, 14:1--24, 2011.
[3]
Y. Ganjisaffar, R. Caruana, and C. V. Lopes. Bagging gradient-boosted trees for high precision, low variance ranking models. SIGIR '11, pages 85--94, 2011.
[4]
M. Hosseini, I. J. Cox, N. Milic-Frayling, G. Kazai, and V. Vinay. On aggregating labels from multiple crowd workers to infer relevance of documents. In ECIR '12.

Cited By

View all
  • (2015)Aggregation of Crowdsourced Ordinal Assessments and Integration with Learning to RankProceedings of the 24th ACM International on Conference on Information and Knowledge Management10.1145/2806416.2806492(1391-1400)Online publication date: 17-Oct-2015

Index Terms

  1. A Modification of LambdaMART to Handle Noisy Crowdsourced Assessments

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Other conferences
      ICTIR '13: Proceedings of the 2013 Conference on the Theory of Information Retrieval
      September 2013
      148 pages
      ISBN:9781450321075
      DOI:10.1145/2499178
      Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

      Sponsors

      • Findwise: Findwise AB
      • Google Inc.
      • Spinque: Spinque
      • Univ. of Copenhagen: University of Copenhagen
      • LARM: LARM Audio Research Archive
      • Royal School of Library and Information Science: Royal School of Library and Information Science
      • Yahoo! Labs

      In-Cooperation

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 29 September 2013

      Check for updates

      Author Tags

      1. Crowdsourcing
      2. Learning-to-Rank
      3. Probabilistic Pairwise Preferences

      Qualifiers

      • Poster
      • Research
      • Refereed limited

      Conference

      ICTIR '13
      Sponsor:
      • Findwise
      • Spinque
      • Univ. of Copenhagen
      • LARM
      • Royal School of Library and Information Science

      Acceptance Rates

      ICTIR '13 Paper Acceptance Rate 11 of 51 submissions, 22%;
      Overall Acceptance Rate 235 of 527 submissions, 45%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)4
      • Downloads (Last 6 weeks)0
      Reflects downloads up to 14 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2015)Aggregation of Crowdsourced Ordinal Assessments and Integration with Learning to RankProceedings of the 24th ACM International on Conference on Information and Knowledge Management10.1145/2806416.2806492(1391-1400)Online publication date: 17-Oct-2015

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media