skip to main content
10.1145/3209978.3210058acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article

From Royals to Vegans: Characterizing Question Trolling on a Community Question Answering Website

Published: 27 June 2018 Publication History

Abstract

The phenomenon of trolling has emerged as a widespread form of abuse on news sites, online social networks, and other types of social media. In this paper, we study a particular type of trolling, performed by asking a provocative question on a community question-answering website. By combining user reports with subsequent moderator deletions, we identify a set of over 400,000 troll questions on Yahoo Answers, i.e., questions aimed to inflame, upset, and draw attention from others on the community. This set of troll questions spans a lengthy period of time and a diverse set of topical categories. Our analysis reveals unique characteristics of troll questions when compared to "regular" questions, with regards to their metadata, text, and askers. A classifier built upon these features reaches an accuracy of 85% over a balanced dataset. The answers' text and metadata, reflecting the community's response to the question, are found particularly productive for the classification task.

References

[1]
Lada A. Adamic, Jun Zhang, Eytan Bakshy, and Mark S. Ackerman . 2008. Knowledge Sharing and Yahoo Answers: Everyone Knows Something Proc. of WWW. 665--674.
[2]
Eugene Agichtein, Carlos Castillo, Debora Donato, Aristides Gionis, and Gilad Mishne . 2008. Finding High-quality Content in Social Media. In Proc. of WSDM. 183--194.
[3]
Antoaneta Baltadzhieva and Grzegorz Chrupala . 2015. Question Quality in Community Question Answering Forums: A Survey. SIGKDD Explor. Newsl. Vol. 17, 1 (2015), 8--13.
[4]
Adam Berger and John Lafferty . 1999. Information Retrieval As Statistical Translation. In Proc. of SIGIR. 222--229.
[5]
Kelly Bergstrom . 2011. “Don't feed the troll”: Shutting down debate about community expectations on Reddit.com. First Monday Vol. 16, 8 (2011).
[6]
Amy Binns . 2012. DON'T FEED THE TROLLS! Managing troublemakers in magazines' online communities. Journalism Practice Vol. 6, 4 (2012), 547--562.
[7]
Steven Bird . 2006. NLTK: The Natural Language Toolkit. In Proc. of COLING-ACL. 69--72.
[8]
Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka . 2017. What Do You Mean Exactly?: Analyzing Clarification Questions in CQA Proc. of CHIIR. 345--348.
[9]
Erin E Buckels, Paul D Trapnell, and Delroy L Paulhus . 2014. Trolls just want to have fun. Personality and individual Differences Vol. 67 (2014), 97--102.
[10]
Erik Cambria, Praphul Chandra, Avinash Sharma, and Amir Hussain . 2010. Do not feel the trolls. Proc. of SDoW Workshop.
[11]
Justin Cheng, Michael Bernstein, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec . 2017. Anyone can become a troll: Causes of trolling behavior in online discussions. arXiv preprint Vol. abs/1702.01119 (2017).
[12]
Justin Cheng, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec . 2015. Antisocial Behavior in Online Discussion Communities Proc. of ICWSM. 61--70.
[13]
Bryn Alexander Coles and Melanie West . 2016. Trolling the Trolls: Online Forum Users Constructions of the Nature and Properties of Trolling. Comput. Hum. Behav. Vol. 60, C (2016), 233--244.
[14]
Sally Jo Cunningham and Annika Hinze . 2014. Social, religious information behavior: An analysis of Yahoo! Answers queries about belief. Advances in the Study of Information and Religion Vol. 4, 1 (2014).
[15]
Imen Ouled Dlala, Dorra Attiaoui, Arnaud Martin, and Boutheina Ben Yaghlane . 2014. Trolls identification within an uncertain framework Proc. of ICTAI. 1011--1015.
[16]
Judith S Donath . 1999. Identity and deception in the virtual community. Communities in cyberspace (1999), 29--59.
[17]
Pnina Fichman and Madelyn Rose Sanfilippo . 2015. The bad boys and girls of cyberspace: How gender and context impact perception of and reaction to trolling. Social science computer review Vol. 33, 2 (2015), 163--180.
[18]
Patxi Galán-Garc'ıa, José Gaviria de la Puerta, Carlos Laorden Gómez, Igor Santos, and Pablo Garc'ıa Bringas . 2016. Supervised machine learning for the detection of troll profiles in twitter social network: Application to a real case of cyberbullying. Logic Journal of the IGPL Vol. 24, 1 (2016), 42--53.
[19]
Rich Gazan . 2016. Seven Words You Can't Say on Answerbag: Contested Terms and Conflict in a Social Q&A Community. In Proc. of HT. 27--36.
[20]
Maja Golf-Papez and Ekant Veer . 2017. Don't feed the trolling: rethinking how online trolling is being defined and combated. Journal of Marketing Management (2017), 1--19.
[21]
Ido Guy, Victor Makarenkov, Niva Hazon, Lior Rokach, and Bracha Shapira . 2018. Identifying Informational vs. Conversational Questions on Community Question Answering Archives. In Proc. of WSDM. 216--224.
[22]
Claire Hardaker . 2010. Trolling in asynchronous computer-mediated communication: From user discussions to academic definitions. Journal of Politeness Research Vol. 6 (2010), 215--242.
[23]
Claire Hardaker . 2015. `I refuse to respond to this obvious troll': an overview of responses to (perceived) trolling. Corpora Vol. 10, 2 (2015), 201--229.
[24]
F. Maxwell Harper, Daniel Moy, and Joseph A. Konstan . 2009. Facts or Friends?: Distinguishing Informational and Conversational Questions in Social Q&A Sites. In Proc. of CHI. 759--768.
[25]
Susan Herring, Kirk Job-Sluder, Rebecca Scheckler, and Sasha Barab . 2002. Searching for safety online: Managing “trolling” in a feminist forum. The Information Society Vol. 18, 5 (2002), 371--384.
[26]
Jaakko Hintikka and Ilpo Halonen . 1995. Semantics and pragmatics for why-questions. The Journal of Philosophy Vol. 92, 12 (1995), 636--657.
[27]
Imrul Kayes, Nicolas Kourtellis, Daniele Quercia, Adriana Iamnitchi, and Francesco Bonchi . 2015. The Social World of Content Abusers in Community Question Answering Proc. of WWW. 570--580.
[28]
Ben Kirman, Conor Lineham, and Shaun Lawson . 2012. Exploring Mischief and Mayhem in Social Computing or: How We Learned to Stop Worrying and Love the Trolls. In Proc. of CHI EA. 121--130.
[29]
April Kontostathis, Kelly Reynolds, Andy Garron, and Lynne Edwards . 2013. Detecting Cyberbullying: Query Terms and Techniques Proc. of WebSci. 195--204.
[30]
Srijan Kumar . 2017. Characterization and Detection of Malicious Behavior on the Web. Ph.D. Dissertation. bibinfoschoolUniversity of Maryland, College Park.
[31]
Jérôme Kunegis, Andreas Lommatzsch, and Christian Bauckhage . 2009. The Slashdot Zoo: Mining a Social Network with Negative Edges Proc. of WWW. 741--750.
[32]
So-Hyun Lee and Hee-Woong Kim . 2015. Why people post benevolent and malicious comments online. Commun. ACM Vol. 58, 11 (2015), 74--79.
[33]
Baichuan Li, Tan Jin, Michael R. Lyu, Irwin King, and Barley Mak . 2012. Analyzing and Predicting Question Quality in Community Question Answering Services Proc. of WWW Companion. 775--782.
[34]
Tai Ching Li, Joobin Gharibshah, Evangelos E Papalexakis, and Michalis Faloutsos . 2017. TrollSpot: Detecting misbehavior in commenting platforms Proc. of ASONAM.
[35]
Lena Mamykina, Bella Manoim, Manas Mittal, George Hripcsak, and Björn Hartmann . 2011. Design Lessons from the Fastest Q&A Site in the West Proc. of CHI. 2857--2866.
[36]
Todor Mihaylov, Georgi Georgiev, and Preslav Nakov . 2015. Finding Opinion Manipulation Trolls in News Community Forums Proc. of CoNLL. 310--314.
[37]
Todor Mihaylov and Preslav Nakov . 2016. Hunting for troll comments in news community forums Proc. of ACL. 399--405.
[38]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean . 2013. Distributed representations of words and phrases and their compositionality Proc. of NIPS. 3111--3119.
[39]
S. Nadali, M. A. A. Murad, N. M. Sharef, A. Mustapha, and S. Shojaee . 2013. A review of cyberbullying detection: An overview. In Proc. of ISDA. 325--330.
[40]
Prankit Namdeo, RK Pateriya, and Sonika Shrivastava . 2017. A Review of Cyber bullying Detection in Social Networking Proc. of ICICCT. 162--170.
[41]
Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang . 2016. Abusive Language Detection in Online User Content. In Proc. of WWW. 145--153.
[42]
F. Javier Ortega, José A. Troyano, Fermin L. Cruz, Carlos G. Vallejo, and Fernando Enriquez . 2012. Propagation of trust and distrust for the detection of trolls in a social network. Computer Networks Vol. 56, 12 (2012), 2884--2895.
[43]
Fabian Pedregosa et al. . 2011. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. Vol. 12 (Nov. . 2011), 2825--2830.
[44]
David J Phillips . 1996. Defending the boundaries: Identifying and countering threats in a Usenet newsgroup. The information society Vol. 12, 1 (1996), 39--62.
[45]
Mattia Samory and Enoch Peserico . 2017. Sizing Up the Troll: A Quantitative Characterization of Moderator-Identified Trolling in an Online Forum. In Proc. of CHI. 6943--6947.
[46]
Chun Wei Seah, Hai Leong Chieu, Kian Ming A Chai, Loo-Nin Teow, and Lee Wei Yeong . 2015. Troll detection by domain-adapting sentiment analysis Proc. of Fusion. 792--799.
[47]
Pnina Shachaf and Noriko Hara . 2010. Beyond Vandalism: Wikipedia Trolls. J. Inf. Sci. Vol. 36, 3 (2010), 357--370.
[48]
Stefan Siersdorfer, Sergiu Chelaru, Jose San Pedro, Ismail Sengor Altingovde, and Wolfgang Nejdl . 2014. Analyzing and Mining Comments and Comment Ratings on the Social Web. ACM Trans. Web Vol. 8, 3 (2014).
[49]
A. Squicciarini, S. Rajtmajer, Y. Liu, and C. Griffin . 2015. Identification and Characterization of Cyberbullying Dynamics in an Online Social Network. In Proc. of ASONAM. 280--285.
[50]
John R Suler and Wende L Phillips . 1998. The bad boys of cyberspace: Deviant behavior in a multimedia chat community. CyberPsychology & Behavior Vol. 1, 3 (1998), 275--294.
[51]
Scott Thacker and Mark D Griffiths . 2012. An exploratory study of trolling in online video gaming. IJCBPL Vol. 2, 4 (2012), 17--33.
[52]
George Tsatsaronis, Filippos Karolos Ventirozos, and Iraklis Varlamis . 2017. Detecting Aggressive Behavior in Discussion Threads Using Text Mining Proc. of CICLing.
[53]
Heidi Vandebosch and Katrien Van Cleemput . 2008. Defining cyberbullying: A qualitative research into the perceptions of youngsters. CyberPsychology & Behavior Vol. 11, 4 (2008), 499--503.
[54]
Ingmar Weber, Antti Ukkonen, and Aris Gionis . 2012. Answers, Not Links: Extracting Tips from Yahoo! Answers to Address How-to Web Queries Proc. of WSDM. 613--622.
[55]
Ryen W. White, Matthew Richardson, and Wen-tau Yih . 2015. Questions vs. Queries in Informational Search Tasks Proc. of WWW Companion. 135--136.
[56]
Zhaoming Wu, Charu C. Aggarwal, and Jimeng Sun . 2016. The Troll-Trust Model for Ranking in Signed Networks Proc. of WSDM. 447--456.

Cited By

View all
  • (2024)The power and potentials of Flexible Query Answering SystemsData & Knowledge Engineering10.1016/j.datak.2023.102246149:COnline publication date: 1-Jan-2024
  • (2023)The Impact of Fake News on Traveling and Antisocial Behavior in Online Communities: OverviewApplied Sciences10.3390/app13211171913:21(11719)Online publication date: 26-Oct-2023
  • (2023)Detecting Covert Disruptive Behavior in Online Interaction by Analyzing Conversational Features and Norm ViolationsACM Transactions on Computer-Human Interaction10.1145/363514331:2(1-43)Online publication date: 1-Dec-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval
June 2018
1509 pages
ISBN:9781450356572
DOI:10.1145/3209978
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 June 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. abusive behavior
  2. antisocial behavior
  3. community question answering
  4. content abuse
  5. question trolling
  6. trolling

Qualifiers

  • Research-article

Conference

SIGIR '18
Sponsor:

Acceptance Rates

SIGIR '18 Paper Acceptance Rate 86 of 409 submissions, 21%;
Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)9
  • Downloads (Last 6 weeks)0
Reflects downloads up to 28 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)The power and potentials of Flexible Query Answering SystemsData & Knowledge Engineering10.1016/j.datak.2023.102246149:COnline publication date: 1-Jan-2024
  • (2023)The Impact of Fake News on Traveling and Antisocial Behavior in Online Communities: OverviewApplied Sciences10.3390/app13211171913:21(11719)Online publication date: 26-Oct-2023
  • (2023)Detecting Covert Disruptive Behavior in Online Interaction by Analyzing Conversational Features and Norm ViolationsACM Transactions on Computer-Human Interaction10.1145/363514331:2(1-43)Online publication date: 1-Dec-2023
  • (2022)“Meating halfway”: Exploring the attitudes of meat eaters, veg*ns, and occasional meat eaters toward those who eat meat and those who do not eat meatThe Journal of Social Psychology10.1080/00224545.2022.2074288163:3(408-424)Online publication date: 10-May-2022
  • (2022)Knowledge map construction for question and answer archivesExpert Systems with Applications: An International Journal10.1016/j.eswa.2019.112923141:COnline publication date: 21-Apr-2022
  • (2021)What identifies different age cohorts in Yahoo! Answers?Knowledge-Based Systems10.1016/j.knosys.2021.107278228:COnline publication date: 23-Aug-2021
  • (2021)The Impact of Question Type and Topic on Misinformation and Trolling on Yahoo! AnswersDiversity, Divergence, Dialogue10.1007/978-3-030-71305-8_10(127-140)Online publication date: 19-Mar-2021
  • (2021)The relationships between misinformation and outrage trolling tactics on two Yahoo! Answers categoriesJournal of the Association for Information Science and Technology10.1002/asi.24497Online publication date: 21-May-2021
  • (2020)Descriptions from the CustomersACM Transactions on Internet Technology10.1145/341820220:4(1-31)Online publication date: 6-Oct-2020
  • (2020)Product Bundle Identification using Semi-Supervised LearningProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401128(791-800)Online publication date: 25-Jul-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media