research-article

From Royals to Vegans: Characterizing Question Trolling on a Community Question Answering Website

Authors:

Bracha ShapiraAuthors Info & Claims

SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval

Pages 835 - 844

https://doi.org/10.1145/3209978.3210058

Published: 27 June 2018 Publication History

Abstract

The phenomenon of trolling has emerged as a widespread form of abuse on news sites, online social networks, and other types of social media. In this paper, we study a particular type of trolling, performed by asking a provocative question on a community question-answering website. By combining user reports with subsequent moderator deletions, we identify a set of over 400,000 troll questions on Yahoo Answers, i.e., questions aimed to inflame, upset, and draw attention from others on the community. This set of troll questions spans a lengthy period of time and a diverse set of topical categories. Our analysis reveals unique characteristics of troll questions when compared to "regular" questions, with regards to their metadata, text, and askers. A classifier built upon these features reaches an accuracy of 85% over a balanced dataset. The answers' text and metadata, reflecting the community's response to the question, are found particularly productive for the classification task.

References

[1]

Lada A. Adamic, Jun Zhang, Eytan Bakshy, and Mark S. Ackerman . 2008. Knowledge Sharing and Yahoo Answers: Everyone Knows Something Proc. of WWW. 665--674.

Digital Library

[2]

Eugene Agichtein, Carlos Castillo, Debora Donato, Aristides Gionis, and Gilad Mishne . 2008. Finding High-quality Content in Social Media. In Proc. of WSDM. 183--194.

Digital Library

[3]

Antoaneta Baltadzhieva and Grzegorz Chrupala . 2015. Question Quality in Community Question Answering Forums: A Survey. SIGKDD Explor. Newsl. Vol. 17, 1 (2015), 8--13.

Digital Library

[4]

Adam Berger and John Lafferty . 1999. Information Retrieval As Statistical Translation. In Proc. of SIGIR. 222--229.

Digital Library

[5]

Kelly Bergstrom . 2011. “Don't feed the troll”: Shutting down debate about community expectations on Reddit.com. First Monday Vol. 16, 8 (2011).

[6]

Amy Binns . 2012. DON'T FEED THE TROLLS! Managing troublemakers in magazines' online communities. Journalism Practice Vol. 6, 4 (2012), 547--562.

[7]

Steven Bird . 2006. NLTK: The Natural Language Toolkit. In Proc. of COLING-ACL. 69--72.

Digital Library

[8]

Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka . 2017. What Do You Mean Exactly?: Analyzing Clarification Questions in CQA Proc. of CHIIR. 345--348.

Digital Library

[9]

Erin E Buckels, Paul D Trapnell, and Delroy L Paulhus . 2014. Trolls just want to have fun. Personality and individual Differences Vol. 67 (2014), 97--102.

[10]

Erik Cambria, Praphul Chandra, Avinash Sharma, and Amir Hussain . 2010. Do not feel the trolls. Proc. of SDoW Workshop.

[11]

Justin Cheng, Michael Bernstein, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec . 2017. Anyone can become a troll: Causes of trolling behavior in online discussions. arXiv preprint Vol. abs/1702.01119 (2017).

Digital Library

[12]

Justin Cheng, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec . 2015. Antisocial Behavior in Online Discussion Communities Proc. of ICWSM. 61--70.

[13]

Bryn Alexander Coles and Melanie West . 2016. Trolling the Trolls: Online Forum Users Constructions of the Nature and Properties of Trolling. Comput. Hum. Behav. Vol. 60, C (2016), 233--244.

Digital Library

[14]

Sally Jo Cunningham and Annika Hinze . 2014. Social, religious information behavior: An analysis of Yahoo! Answers queries about belief. Advances in the Study of Information and Religion Vol. 4, 1 (2014).

[15]

Imen Ouled Dlala, Dorra Attiaoui, Arnaud Martin, and Boutheina Ben Yaghlane . 2014. Trolls identification within an uncertain framework Proc. of ICTAI. 1011--1015.

Digital Library

[16]

Judith S Donath . 1999. Identity and deception in the virtual community. Communities in cyberspace (1999), 29--59.

[17]

Pnina Fichman and Madelyn Rose Sanfilippo . 2015. The bad boys and girls of cyberspace: How gender and context impact perception of and reaction to trolling. Social science computer review Vol. 33, 2 (2015), 163--180.

Digital Library

[18]

Patxi Galán-Garc'ıa, José Gaviria de la Puerta, Carlos Laorden Gómez, Igor Santos, and Pablo Garc'ıa Bringas . 2016. Supervised machine learning for the detection of troll profiles in twitter social network: Application to a real case of cyberbullying. Logic Journal of the IGPL Vol. 24, 1 (2016), 42--53.

[19]

Rich Gazan . 2016. Seven Words You Can't Say on Answerbag: Contested Terms and Conflict in a Social Q&A Community. In Proc. of HT. 27--36.

Digital Library

[20]

Maja Golf-Papez and Ekant Veer . 2017. Don't feed the trolling: rethinking how online trolling is being defined and combated. Journal of Marketing Management (2017), 1--19.

[21]

Ido Guy, Victor Makarenkov, Niva Hazon, Lior Rokach, and Bracha Shapira . 2018. Identifying Informational vs. Conversational Questions on Community Question Answering Archives. In Proc. of WSDM. 216--224.

Digital Library

[22]

Claire Hardaker . 2010. Trolling in asynchronous computer-mediated communication: From user discussions to academic definitions. Journal of Politeness Research Vol. 6 (2010), 215--242.

[23]

Claire Hardaker . 2015. `I refuse to respond to this obvious troll': an overview of responses to (perceived) trolling. Corpora Vol. 10, 2 (2015), 201--229.

[24]

F. Maxwell Harper, Daniel Moy, and Joseph A. Konstan . 2009. Facts or Friends?: Distinguishing Informational and Conversational Questions in Social Q&A Sites. In Proc. of CHI. 759--768.

Digital Library

[25]

Susan Herring, Kirk Job-Sluder, Rebecca Scheckler, and Sasha Barab . 2002. Searching for safety online: Managing “trolling” in a feminist forum. The Information Society Vol. 18, 5 (2002), 371--384.

[26]

Jaakko Hintikka and Ilpo Halonen . 1995. Semantics and pragmatics for why-questions. The Journal of Philosophy Vol. 92, 12 (1995), 636--657.

[27]

Imrul Kayes, Nicolas Kourtellis, Daniele Quercia, Adriana Iamnitchi, and Francesco Bonchi . 2015. The Social World of Content Abusers in Community Question Answering Proc. of WWW. 570--580.

Digital Library

[28]

Ben Kirman, Conor Lineham, and Shaun Lawson . 2012. Exploring Mischief and Mayhem in Social Computing or: How We Learned to Stop Worrying and Love the Trolls. In Proc. of CHI EA. 121--130.

Digital Library

[29]

April Kontostathis, Kelly Reynolds, Andy Garron, and Lynne Edwards . 2013. Detecting Cyberbullying: Query Terms and Techniques Proc. of WebSci. 195--204.

Digital Library

[30]

Srijan Kumar . 2017. Characterization and Detection of Malicious Behavior on the Web. Ph.D. Dissertation. bibinfoschoolUniversity of Maryland, College Park.

[31]

Jérôme Kunegis, Andreas Lommatzsch, and Christian Bauckhage . 2009. The Slashdot Zoo: Mining a Social Network with Negative Edges Proc. of WWW. 741--750.

Digital Library

[32]

So-Hyun Lee and Hee-Woong Kim . 2015. Why people post benevolent and malicious comments online. Commun. ACM Vol. 58, 11 (2015), 74--79.

Digital Library

[33]

Baichuan Li, Tan Jin, Michael R. Lyu, Irwin King, and Barley Mak . 2012. Analyzing and Predicting Question Quality in Community Question Answering Services Proc. of WWW Companion. 775--782.

Digital Library

[34]

Tai Ching Li, Joobin Gharibshah, Evangelos E Papalexakis, and Michalis Faloutsos . 2017. TrollSpot: Detecting misbehavior in commenting platforms Proc. of ASONAM.

Digital Library

[35]

Lena Mamykina, Bella Manoim, Manas Mittal, George Hripcsak, and Björn Hartmann . 2011. Design Lessons from the Fastest Q&A Site in the West Proc. of CHI. 2857--2866.

Digital Library

[36]

Todor Mihaylov, Georgi Georgiev, and Preslav Nakov . 2015. Finding Opinion Manipulation Trolls in News Community Forums Proc. of CoNLL. 310--314.

[37]

Todor Mihaylov and Preslav Nakov . 2016. Hunting for troll comments in news community forums Proc. of ACL. 399--405.

[38]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean . 2013. Distributed representations of words and phrases and their compositionality Proc. of NIPS. 3111--3119.

Digital Library

[39]

S. Nadali, M. A. A. Murad, N. M. Sharef, A. Mustapha, and S. Shojaee . 2013. A review of cyberbullying detection: An overview. In Proc. of ISDA. 325--330.

[40]

Prankit Namdeo, RK Pateriya, and Sonika Shrivastava . 2017. A Review of Cyber bullying Detection in Social Networking Proc. of ICICCT. 162--170.

[41]

Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang . 2016. Abusive Language Detection in Online User Content. In Proc. of WWW. 145--153.

Digital Library

[42]

F. Javier Ortega, José A. Troyano, Fermin L. Cruz, Carlos G. Vallejo, and Fernando Enriquez . 2012. Propagation of trust and distrust for the detection of trolls in a social network. Computer Networks Vol. 56, 12 (2012), 2884--2895.

Digital Library

[43]

Fabian Pedregosa et al. . 2011. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. Vol. 12 (Nov. . 2011), 2825--2830.

Digital Library

[44]

David J Phillips . 1996. Defending the boundaries: Identifying and countering threats in a Usenet newsgroup. The information society Vol. 12, 1 (1996), 39--62.

[45]

Mattia Samory and Enoch Peserico . 2017. Sizing Up the Troll: A Quantitative Characterization of Moderator-Identified Trolling in an Online Forum. In Proc. of CHI. 6943--6947.

Digital Library

[46]

Chun Wei Seah, Hai Leong Chieu, Kian Ming A Chai, Loo-Nin Teow, and Lee Wei Yeong . 2015. Troll detection by domain-adapting sentiment analysis Proc. of Fusion. 792--799.

[47]

Pnina Shachaf and Noriko Hara . 2010. Beyond Vandalism: Wikipedia Trolls. J. Inf. Sci. Vol. 36, 3 (2010), 357--370.

Digital Library

[48]

Stefan Siersdorfer, Sergiu Chelaru, Jose San Pedro, Ismail Sengor Altingovde, and Wolfgang Nejdl . 2014. Analyzing and Mining Comments and Comment Ratings on the Social Web. ACM Trans. Web Vol. 8, 3 (2014).

Digital Library

[49]

A. Squicciarini, S. Rajtmajer, Y. Liu, and C. Griffin . 2015. Identification and Characterization of Cyberbullying Dynamics in an Online Social Network. In Proc. of ASONAM. 280--285.

Digital Library

[50]

John R Suler and Wende L Phillips . 1998. The bad boys of cyberspace: Deviant behavior in a multimedia chat community. CyberPsychology & Behavior Vol. 1, 3 (1998), 275--294.

[51]

Scott Thacker and Mark D Griffiths . 2012. An exploratory study of trolling in online video gaming. IJCBPL Vol. 2, 4 (2012), 17--33.

Digital Library

[52]

George Tsatsaronis, Filippos Karolos Ventirozos, and Iraklis Varlamis . 2017. Detecting Aggressive Behavior in Discussion Threads Using Text Mining Proc. of CICLing.

[53]

Heidi Vandebosch and Katrien Van Cleemput . 2008. Defining cyberbullying: A qualitative research into the perceptions of youngsters. CyberPsychology & Behavior Vol. 11, 4 (2008), 499--503.

[54]

Ingmar Weber, Antti Ukkonen, and Aris Gionis . 2012. Answers, Not Links: Extracting Tips from Yahoo! Answers to Address How-to Web Queries Proc. of WSDM. 613--622.

Digital Library

[55]

Ryen W. White, Matthew Richardson, and Wen-tau Yih . 2015. Questions vs. Queries in Informational Search Tasks Proc. of WWW Companion. 135--136.

Digital Library

[56]

Zhaoming Wu, Charu C. Aggarwal, and Jimeng Sun . 2016. The Troll-Trust Model for Ranking in Signed Networks Proc. of WSDM. 447--456.

Digital Library

Cited By

Andreasen TBordogna GTré GKacprzyk JLarsen HZadrożny S(2024)The power and potentials of Flexible Query Answering SystemsData & Knowledge Engineering10.1016/j.datak.2023.102246149:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.datak.2023.102246
Stupavský IDakić PVranić V(2023)The Impact of Fake News on Traveling and Antisocial Behavior in Online Communities: OverviewApplied Sciences10.3390/app13211171913:21(11719)Online publication date: 26-Oct-2023
https://doi.org/10.3390/app132111719
Paakki HVepsäläinen HSalovaara AZafar B(2023)Detecting Covert Disruptive Behavior in Online Interaction by Analyzing Conversational Features and Norm ViolationsACM Transactions on Computer-Human Interaction10.1145/363514331:2(1-43)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1145/3635143
Show More Cited By

Index Terms

From Royals to Vegans: Characterizing Question Trolling on a Community Question Answering Website
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
      1. Social media
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Question answering
  2. World Wide Web
    1. Web applications
      1. Crowdsourcing

Recommendations

Anyone Can Become a Troll: Causes of Trolling Behavior in Online Discussions
CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

In online communities, antisocial behavior such as trolling disrupts constructive discussion. While prior work suggests that trolling behavior is confined to a vocal and antisocial minority, we demonstrate that ordinary people can engage in such behavior ...
Anonymity, User Engagement, Quality, and Trolling on Q&A Sites
CSCW

In online question and answer (Q&A) communities, people ask questions and share answers at all levels of topic sensitivity. Identity options within these communities range from anonymity to real name. The amount of engagement, and the quality of ...
Community Interaction and Conflict on the Web
WWW '18: Proceedings of the 2018 World Wide Web Conference

Users organize themselves into communities on web platforms. These communities can interact with one another, often leading to conflicts and toxic interactions. However, little is known about the mechanisms of interactions between communities and how ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval

June 2018

1509 pages

ISBN:9781450356572

DOI:10.1145/3209978

General Chairs:
Kevyn Collins-Thompson
University of Michigan, United States
,
Qiaozhu Mei
University of Michigan, United States
,
Program Chairs:
Brian Davison
Lehigh University, United States
,
Yiqun Liu
Tsinghua University, China
,
Emine Yilmaz
University College London, United Kingdom

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR '18

Sponsor:

SIGIR

SIGIR '18: The 41st International ACM SIGIR conference on research and development in Information Retrieval

July 8 - 12, 2018

MI, Ann Arbor, USA

Acceptance Rates

SIGIR '18 Paper Acceptance Rate 86 of 409 submissions, 21%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
453
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Andreasen TBordogna GTré GKacprzyk JLarsen HZadrożny S(2024)The power and potentials of Flexible Query Answering SystemsData & Knowledge Engineering10.1016/j.datak.2023.102246149:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.datak.2023.102246
Stupavský IDakić PVranić V(2023)The Impact of Fake News on Traveling and Antisocial Behavior in Online Communities: OverviewApplied Sciences10.3390/app13211171913:21(11719)Online publication date: 26-Oct-2023
https://doi.org/10.3390/app132111719
Paakki HVepsäläinen HSalovaara AZafar B(2023)Detecting Covert Disruptive Behavior in Online Interaction by Analyzing Conversational Features and Norm ViolationsACM Transactions on Computer-Human Interaction10.1145/363514331:2(1-43)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1145/3635143
Pabian SOuvrein Gvan Royen KFolkvord FPoels KVandebosch HDe Backer C(2022)“Meating halfway”: Exploring the attitudes of meat eaters, veg*ns, and occasional meat eaters toward those who eat meat and those who do not eat meatThe Journal of Social Psychology10.1080/00224545.2022.2074288163:3(408-424)Online publication date: 10-May-2022
https://doi.org/10.1080/00224545.2022.2074288
Li MLu XChen LWang J(2022)Knowledge map construction for question and answer archivesExpert Systems with Applications: An International Journal10.1016/j.eswa.2019.112923141:COnline publication date: 21-Apr-2022
https://dl.acm.org/doi/10.1016/j.eswa.2019.112923
Figueroa ATimilsina M(2021)What identifies different age cohorts in Yahoo! Answers?Knowledge-Based Systems10.1016/j.knosys.2021.107278228:COnline publication date: 23-Aug-2021
https://dl.acm.org/doi/10.1016/j.knosys.2021.107278
Fichman PBrill R(2021)The Impact of Question Type and Topic on Misinformation and Trolling on Yahoo! AnswersDiversity, Divergence, Dialogue10.1007/978-3-030-71305-8_10(127-140)Online publication date: 19-Mar-2021
https://doi.org/10.1007/978-3-030-71305-8_10
Fichman PVaughn M(2021)The relationships between misinformation and outrage trolling tactics on two Yahoo! Answers categoriesJournal of the Association for Information Science and Technology10.1002/asi.24497Online publication date: 21-May-2021
https://doi.org/10.1002/asi.24497
Novgorodov SGuy IElad GRadinsky K(2020)Descriptions from the CustomersACM Transactions on Internet Technology10.1145/341820220:4(1-31)Online publication date: 6-Oct-2020
https://dl.acm.org/doi/10.1145/3418202
Tzaban HGuy IGreenstein-Messica ADagan ARokach LShapira BHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)Product Bundle Identification using Semi-Supervised LearningProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401128(791-800)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401128
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten