Skip to main content
SpringerLink
Log in

Best and Fairest: An Empirical Analysis of Retrieval System Bias

  • Conference paper
Advances in Information Retrieval (ECIR 2014)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 8416))

Included in the following conference series:

Abstract

In this paper, we explore the bias of term weighting schemes used by retrieval models. Here, we consider bias as the extent to which a retrieval model unduly favours certain documents over others because of characteristics within and about the document. We set out to find the least biased retrieval model/weighting. This is largely motivated by the recent proposal of a new suite of retrieval models based on the Divergence From Independence (DFI) framework. The claim is that such models provide the fairest term weighting because they do not make assumptions about the term distribution (unlike most other retrieval models). In this paper, we empirically examine whether fairness is linked to performance and answer the question; is fairer better?

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Amati, G., Van Rijsbergen, C.J.: Probabilistic models of ir based on measuring the divergence from randomness. ACM Trans. on Info. Sys., 357–389 (2002)

    Google Scholar 

  2. Azzopardi, L., Bache, R.: On the relationship between effectiveness and accessibility. In: Proc. of the 33rd international ACM SIGIR, pp. 889–890 (2010)

    Google Scholar 

  3. Azzopardi, L., Vinay, V.: Retrievability: An evaluation measure for higher order information access tasks. In: Proc. of the 17th ACM CIKM, pp. 561–570 (2008)

    Google Scholar 

  4. Bashir, S., Rauber, A.: On the relationship bw query characteristics and ir functions retrieval bias. J. Am. Soc. Inf. Sci. Technol. 62(8), 1515–1532 (2011)

    Article  Google Scholar 

  5. Clinchant, S., Gaussier, E.: Bridging language modeling and divergence from randomness models: A log-logistic model for ir. In: Azzopardi, L., Kazai, G., Robertson, S., Rüger, S., Shokouhi, M., Song, D., Yilmaz, E. (eds.) ICTIR 2009. LNCS, vol. 5766, pp. 54–65. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  6. Crestani, F., Lalmas, M., Van Rijsbergen, C.J., Campbell, I.: Is this document relevant? probably: a survey of probabilistic models in information retrieval. ACM Computing Survey 30(4), 528–552 (1998)

    Article  Google Scholar 

  7. Dinçer, B.T., Kocabas, I., Karaoglan, B.: Irra at trec 2010: Index term weighting by divergence from independence model. In: TREC (2010)

    Google Scholar 

  8. Fang, H., Tao, T., Zhai, C.: A formal study of information retrieval heuristics. In: Proc. of the 27th ACM SIGIR Conference, SIGIR 2004, pp. 49–56 (2004)

    Google Scholar 

  9. Fuhr, N.: Probabilistic models in ir. Computer Journal 35(3), 243–255 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  10. Gastwirth, J.: The estimation of the lorenz curve and gini index. The Review of Economics and Statistics 54, 306–316 (1972)

    Article  MathSciNet  Google Scholar 

  11. Harter, S.P.: A probabilistic approach to automatic keyword indexing. part i. on the distribution of specialty words in a technical literature. Journal of the American Society for Information Science 26(4), 197–206 (1975)

    Article  Google Scholar 

  12. Hiemstra, D.: A probabilistic justification for using tf.idf term weighting in information retrieval. International Journal on Digital Libraries 3(2), 131–139 (2000)

    Article  Google Scholar 

  13. Jones, K.S., Walker, S., Robertson, S.E.: A probabilistic model of information retrieval: development and comparative experiments (parts 1 and 2). Information Processing and Management 36(6), 779–808 (2000)

    Article  Google Scholar 

  14. Kocabas, I., Dinçer, B.T., Karaoglan, B.: A nonparametric term weighting method for information retrieval based on measuring the divergence from independence. Information Retrieval, 1–24 (2013)

    Google Scholar 

  15. Losada, D.E., Azzopardi, L., Baillie, M.: Revisiting the relationship between doc. length and relevance. In: Proc. of the 17th ACM CIKM 2008, pp. 419–428 (2008)

    Google Scholar 

  16. Maron, M.E., Kuhns, J.L.: On relevance, probabilistic indexing and information retrieval. Journal of the ACM 7(3), 216–244 (1960)

    Article  Google Scholar 

  17. Ponte, J.M., Croft, W.B.: A language modeling approach to information retrieval. In: Proc. of the 21st ACM SIGIR Conference, SIGIR 1998, pp. 275–281 (1998)

    Google Scholar 

  18. Robertson, S.E., Walker, S.: Some simple effective approx. to the 2-poisson model for probabilistic weighted retrieval. In: Proc. of ACM SIGIR 1994, pp. 232–241 (1994)

    Google Scholar 

  19. Salton, G., Wong, A., Yang, C.S.: A vector space model for automatic indexing. Communications of the ACM 18(11), 613–620 (1975)

    Article  MATH  Google Scholar 

  20. Salton, G.: Automatic Information Organization and Retrieval (1968)

    Google Scholar 

  21. Singhal, A., Buckley, C., Mitra, M.: Pivoted document length normalization. In: Proce. of the 19th ACM SIGIR Conference, SIGIR 1996, pp. 21–29 (1996)

    Google Scholar 

  22. Wilkie, C., Azzopardi, L.: Relating retrievability, performance and length. In: Proc. of the 36th ACM SIGIR Conference, SIGIR 2013, pp. 937–940 (2013)

    Google Scholar 

  23. Zhai, C., Lafferty, J.: A study of smoothing methods for language models applied to ad hoc ir. In: Proc. of the 24th ACM SIGIR, pp. 334–342 (2001)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing Science, University of Glasgow, Scotland, UK

    Colin Wilkie & Leif Azzopardi

Authors
  1. Colin Wilkie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leif Azzopardi
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University of Amsterdam, Amsterdam, The Netherlands

    Maarten de Rijke  & Tom Kenter  & 

  2. Centrum Wiskunde en Informatica, Amsterdam, The Netherlands and Delft University of Technology, Delft, The Netherlands

    Arjen P. de Vries

  3. University of Illinois at Urbana-Champaign, Urbana, IL, USA

    ChengXiang Zhai

  4. University of Twente, Twente, The Netheralnds and Erasmus University Rotterdam, Rotterdam, The Netherlands

    Franciska de Jong

  5. SalesPredict, Haifa, Israel

    Kira Radinsky

  6. Microsoft Research, Cambridge, UK

    Katja Hofmann

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this paper

Cite this paper

Wilkie, C., Azzopardi, L. (2014). Best and Fairest: An Empirical Analysis of Retrieval System Bias. In: de Rijke, M., et al. Advances in Information Retrieval. ECIR 2014. Lecture Notes in Computer Science, vol 8416. Springer, Cham. https://doi.org/10.1007/978-3-319-06028-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-06028-6_2

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-06027-9

  • Online ISBN: 978-3-319-06028-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation