skip to main content
10.1145/3630744.3658415acmconferencesArticle/Chapter ViewAbstractPublication PageswebsciConference Proceedingsconference-collections
extended-abstract

Investigating Bias in Political Search Query Suggestions by Relative Comparison with LLMs

Published: 13 June 2024 Publication History

Abstract

Search query suggestions affect users’ interactions with search engines, which then influences the information they encounter. Thus, bias in search query suggestions can lead to exposure to biased search results and can impact opinion formation. This is especially critical in the political domain. Detecting and quantifying bias in web search engines is difficult due to its topic dependency, complexity, and subjectivity. The lack of context and phrasality of query suggestions emphasizes this problem. In a multi-step approach, we combine the benefits of large language models, pairwise comparison, and Elo-based scoring to identify and quantify bias in English search query suggestions. We apply our approach to the U.S. political news domain and compare bias in Google and Bing.

References

[1]
AllSides. 2021. How AllSides Creates Balanced News: A Step-by-Step Guide. Retrieved Nov 30, 2022 from allsides.com/blog/how-does-allsides-create-balanced-news
[2]
M. Bonart, A. Samokhina, G. Heisenberg, and P. Schaer. 2019. An investigation of biases in web search engine query suggestions. OIR 44, 2 (2019), 365–381.
[3]
M. Boubdir, E. Kim, B. Ermis, S. Hooker, and M. Fadaee. 2023. Elo Uncovered: Robustness and Best Practices in Language Model Evaluation. arxiv:2311.17295 [cs.CL]
[4]
Wei-Fan Chen, Khalid Al Khatib, Henning Wachsmuth, and Benno Stein. 2020. Analyzing Political Bias and Unfairness in News Articles at Different Levels of Granularity. In NLPCSS. 149–154. https://doi.org/10.18653/v1/2020.nlpcss-1.16
[5]
R. Epstein and R.E. Robertson. 2015. The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. PNAS 112, 33, E4512–E4521.
[6]
Bertram Gawronski. 2021. Partisan bias in the identification of fake news. TiCS 25, 9 (2021), 723–724.
[7]
I. J. Good. 1955. On the Marking of Chess-Players. The Mathematical Gazette 39, 330 (1955), 292–296. http://www.jstor.org/stable/3608567
[8]
F. Haak and P. Schaer. 2021. Perception-Aware Bias Detection for Query Suggestions. In BIAS. 130–142.
[9]
F. Haak and P. Schaer. 2022. Auditing Search Query Suggestion Bias Through Recursive Algorithm Interrogation. In WebSci ’22. 219–227.
[10]
F. Haak and P. Schaer. 2023. Qbias - A Dataset on Media Bias in Search Queries and Query Suggestions. In WebSci ’23. 239–244.
[11]
Christoph Hube and Besnik Fetahu. 2019. Neural Based Statement Classification for Biased Language. In WSDM. ACM.
[12]
L. Introna and H. Nissenbaum. 2000. Defining the Web: the politics of search engines. Computer 33, 1, 54–62.
[13]
J. Kulshrestha, M. Eslami, J. Messias, M. B. Zafar, S. Ghosh, K. P. Gummadi, and K. Karahalios. 2019. Search bias quantification: investigating political bias in social media and web search. Inf. Retr. J. 22, 1, 188–227.
[14]
Ruibo Liu, Chenyan Jia, and Soroush Vosoughi. 2021. A Transformer-based Framework for Neutralizing and Reversing the Political Polarity of News Articles. Proc. ACM Hum.-Comput. Interact. 5, 1–26.
[15]
Negar Mokhberian, André s Abeliuk, Patrick Cummings, and Kristina Lerman. 2020. Moral Framing and Ideological Bias of News. In SocInfo. 206–219.
[16]
Xi Niu and Diane Kelly. 2014. The use of query suggestions during information search. IPM 50, 1, 218–234.
[17]
L. Ray. 2020. 2020 Google Search Survey: How Much Do Users Trust Their Search Results?Retrieved Nov 30, 2022 from moz.com/blog/2020-google-search-survey
[18]
Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic Models for Analyzing and Detecting Biased Language. In ACL. 1650–1659.
[19]
R. E. Robertson, S. Jiang, K. Joseph, L. Friedland, D. Lazer, and C. Wilson. 2018. Auditing Partisan Audience Bias within Google Search. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 148 (2018).
[20]
T. Spinde, S. Hinterreiter, F. Haak, T. Ruas, H. Giese, N. Meuschke, and B. Gipp. 2023. The Media Bias Taxonomy: A Systematic Literature Review on the Forms and Automated Detection of Media Bias. (2023). arxiv:2312.16148 [cs.CL]

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
Websci Companion '24: Companion Publication of the 16th ACM Web Science Conference
May 2024
128 pages
ISBN:9798400704536
DOI:10.1145/3630744
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2024

Check for updates

Author Tags

  1. bias
  2. large language model
  3. pairwise comparison
  4. query suggestion
  5. search queries
  6. web search

Qualifiers

  • Extended-abstract
  • Research
  • Refereed limited

Conference

Websci '24
Sponsor:
Websci '24: 16th ACM Web Science Conference
May 21 - 24, 2024
Stuttgart, Germany

Acceptance Rates

Websci Companion '24 Paper Acceptance Rate 27 of 58 submissions, 47%;
Overall Acceptance Rate 245 of 933 submissions, 26%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 46
    Total Downloads
  • Downloads (Last 12 months)46
  • Downloads (Last 6 weeks)8
Reflects downloads up to 27 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media