skip to main content
10.1145/3630744.3658412acmconferencesArticle/Chapter ViewAbstractPublication PageswebsciConference Proceedingsconference-collections
extended-abstract

Who is targeted? Detecting social group mentions in online political discussions

Published: 13 June 2024 Publication History

Abstract

Social groups are central to political discussions. However, detecting social groups in text often relies on pre-determined socio-demographic categories or supervised learning methods that require extensive hand-labeled datasets. In this paper, we propose a methodology designed to leverage the potential of Large Language Models (LLMs) for the identification and annotation of social groups in text. The experiments show that open LLMs like Llama-2-70B-Chat and Mixtral-8-7B can reliably be used to annotate social groups in a few-shot scenario without the need for supervised learning. The automatically obtained annotations largely match human annotations on random samples from the Reddit Politosphere, resulting in micro-F1 scores of 0.71 and 0.83, respectively.

References

[1]
D. R. Forsyth. 2014. Group dynamics (6th ed.). Wadsworth Cengage Learning, Belmont, CA. OCLC: ocn826872491.
[2]
F. Gilardi, M. Alizadeh, and M. Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks. Proc. Natl. Acad. Sci. U.S.A. 120, 30 (2023).
[3]
V. Hofmann, H. Schütze, and Janet B. Pierrehumbert. 2022. The Reddit Politosphere: A Large-Scale Text and Network Resource of Online Political Discourse. ICWSM 2022 16 (2022), 1259–1267.
[4]
H. Licht and R. Sczepanski. 2023. Who are they talking about? Detecting mentions of social groups in political texts with supervised learning.
[5]
M. Thau. 2019. How Political Parties Use Group-Based Appeals: Evidence from Britain 1964–2015. Political Studies 67, 1 (2019), 63–82.
[6]
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. Smith, R. Subramanian, X. Tan, B. Tang, R. Taylor, A. Williams, J. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 [cs].
[7]
S. Wang, X. Sun, X. Li, R. Ouyang, F. Wu, T. Zhang, J. Li, and G. Wang. 2023. GPT-NER: Named Entity Recognition via Large Language Models. arXiv:2304.10428 [cs].
[8]
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs].

Index Terms

  1. Who is targeted? Detecting social group mentions in online political discussions

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    Websci Companion '24: Companion Publication of the 16th ACM Web Science Conference
    May 2024
    128 pages
    ISBN:9798400704536
    DOI:10.1145/3630744
    Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 13 June 2024

    Check for updates

    Author Tags

    1. LLMs
    2. political text
    3. social groups

    Qualifiers

    • Extended-abstract
    • Research
    • Refereed limited

    Funding Sources

    • Ministry of Science, Research and the Arts Baden-Württemberg, Germany

    Conference

    Websci '24
    Sponsor:
    Websci '24: 16th ACM Web Science Conference
    May 21 - 24, 2024
    Stuttgart, Germany

    Acceptance Rates

    Websci Companion '24 Paper Acceptance Rate 27 of 58 submissions, 47%;
    Overall Acceptance Rate 245 of 933 submissions, 26%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 71
      Total Downloads
    • Downloads (Last 12 months)71
    • Downloads (Last 6 weeks)8
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media