research-article

Investigating Toxicity Across Multiple Reddit Communities, Users, and Moderators

Authors:

Hind Almerekhi,

Supervised by Bernard J. Jansen,

co-supervised by Haewoon KwakAuthors Info & Claims

WWW '20: Companion Proceedings of the Web Conference 2020

Pages 294 - 298

https://doi.org/10.1145/3366424.3382091

Published: 20 April 2020 Publication History

Abstract

Online platforms like Reddit enable users to build communities and converse about diverse topics and interests. However, with the increasing number of users that post disturbing comments containing profanity, harassment, and hate speech, otherwise known as toxic comments. Moderators often struggle with managing the safety of discussions in online communities. To address these issues, we need to detect toxic comments and the root causes of toxicity in discussion threads, i.e., toxicity triggers. Additionally, we need to investigate the toxic posting behavior of users to understand how it differs across online communities and consolidate our findings with moderators from Reddit. In this work, we present our approach, which builds on state-of-the-art methods of toxic comment and toxicity trigger detection. Lastly, we present our research findings of investigating toxicity across users and moderators on Reddit.

References

[1]

Hind Almerekhi, Haewoon Kwak, and Bernard J. Jansen. 2020. Statistical Modeling of Harassment against Reddit Moderators. In Companion Proceedings of the Web Conference 2020 (Taipei, Taiwan) (WWW ’20). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE.

Digital Library

[2]

Hind Almerekhi, Haewoon Kwak, Bernard J. Jansen, and Joni Salminen. 2019. Detecting Toxicity Triggers in Online Discussions. In Proceedings of the 30th ACM Conference on Hypertext and Social Media (Hof, Germany) (HT ’19). Association for Computing Machinery, New York, NY, USA, 291–292.

Digital Library

[3]

Hind Almerekhi, Haewoon Kwak, Joni Salminen, and Bernard J. Jansen. 2020. Are These Comments Triggering? Predicting Triggers of Toxicity in Online Discussions. In Proceedings of the 29th International Conference on World Wide Web (Taipei, Taiwan) (WWW ’20). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE.

Digital Library

[4]

Lora Aroyo, Lucas Dixon, Nithum Thain, Olivia Redfield, and Rachel Rosen. 2019. Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions. In Companion Proceedings of The 2019 World Wide Web Conference (San Francisco, USA) (WWW ’19). Association for Computing Machinery, New York, NY, USA, 1100–1105.

Digital Library

[5]

Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep Learning for Hate Speech Detection in Tweets. In Proceedings of the 26th International Conference on World Wide Web Companion (Perth, Australia) (WWW ’17 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 759–760.

[6]

Koray Balci and Albert Ali Salah. 2015. Automatic analysis and identification of verbal aggression and abusive behaviors for online social games. Computers in Human Behavior 53 (2015), 517–526.

Digital Library

[7]

Eshwar Chandrasekharan, Mattia Samory, Anirudh Srinivasan, and Eric Gilbert. 2017. The Bag of Communities: Identifying Abusive Behavior Online with Preexisting Internet Data. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 3175–3187.

Digital Library

[8]

Jonathan Chang and Cristian Danescu-Niculescu-Mizil. 2019. Trajectories of Blocked Community Members: Redemption, Recidivism and Departure. In The World Wide Web Conference (San Francisco, CA, USA) (WWW ’19). Association for Computing Machinery, New York, NY, USA, 184–195.

Digital Library

[9]

Justin Cheng, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec. 2015. Antisocial Behavior in Online Discussion Communities.

[10]

Maeve Duggan. 2017. Online harassment 2017. (2017).

[11]

Spiros V. Georgakopoulos, Sotiris K. Tasoulis, Aristidis G. Vrahatis, and Vassilis P. Plagianakos. 2018. Convolutional Neural Networks for Toxic Comment Classification. In Proceedings of the 10th Hellenic Conference on Artificial Intelligence (Patras, Greece) (SETN ’18). Association for Computing Machinery, New York, NY, USA, Article Article 35, 6 pages.

Digital Library

[12]

Anna Gibson. 2019. Free Speech and Safe Spaces: How Moderation Policies Shape Online Discussion Spaces. Social Media + Society 5, 1 (2019), 2056305119832588.

[13]

Lars Kai Hansen, Adam Arvidsson, Finn Aarup Nielsen, Elanor Colleoni, and Michael Etter. 2011. Good Friends, Bad News - Affect and Virality in Twitter. In Future Information Technology, James J. Park, Laurence T. Yang, and Changhoon Lee(Eds.). Springer, Berlin, Heidelberg, 34–43.

[14]

Florian Hauser, Julia Hautz, Katja Hutter, and Johann Füller. 2017. Firestorms: Modeling conflict diffusion and management strategies in online communities. The Journal of Strategic Information Systems 26, 4 (2017), 285 – 321.

Digital Library

[15]

Xia Hu, Jiliang Tang, Yanchao Zhang, and Huan Liu. 2013. Social spammer detection in microblogging. In Twenty-Third International Joint Conference on Artificial Intelligence.

Digital Library

[16]

Prerna Juneja, Deepika Rama Subramanian, and Tanushree Mitra. 2020. Through the Looking Glass: Study of Transparency in Reddit’s Moderation Practices. Proc. ACM Hum.-Comput. Interact. 4, GROUP, Article Article 17 (Jan. 2020), 35 pages.

Digital Library

[17]

Haewoon Kwak, Jeremy Blackburn, and Seungyeop Han. 2015. Exploring cyberbullying and other toxic behavior in team competition online games. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 3739–3748.

Digital Library

[18]

Cheng-Yu Lai and Chia-Hua Tsai. 2016. Cyberbullying in the Social Networking Sites: An Online Disinhibition Effect Perspective. In Proceedings of the The 3rd Multidisciplinary International Social Networks Conference on SocialInformatics 2016, Data Science 2016 (Union, NJ, USA) (MISNC, SI, DS 2016). Association for Computing Machinery, New York, NY, USA, Article Article 4, 6 pages.

Digital Library

[19]

Jo Lander. 2015. Building community in online discussion: A case study of moderator strategies. Linguistics and Education 29 (2015), 107 – 120.

[20]

Chenyi Lei, Shouling Ji, and Zhao Li. 2019. TiSSA: A Time Slice Self-Attention Approach for Modeling Sequential User Behaviors. In The World Wide Web Conference (San Francisco, CA, USA) (WWW ’19). Association for Computing Machinery, New York, NY, USA, 2964–2970.

[21]

U. Matzat and G. Rooks. 2014. Styles of moderation in online health and support communities: An experimental comparison of their acceptance and effectiveness. Computers in Human Behavior 36 (2014), 65 – 75.

Digital Library

[22]

Joaquim A. M. Neto, Kazuki M. Yokoyama, and Karin Becker. 2017. Studying Toxic Behavior Influence and Player Chat in an Online Video Game. In Proceedings of the International Conference on Web Intelligence (Leipzig, Germany) (WI ’17). Association for Computing Machinery, New York, NY, USA, 26–33.

Digital Library

[23]

Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive Language Detection in Online User Content. In Proceedings of the 25th International Conference on World Wide Web (Montréal, Québec, Canada) (WWW ’16). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 145–153.

[24]

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532–1543.

[25]

Joseph Seering, Tony Wang, Jina Yoon, and Geoff Kaufman. 2019. Moderator engagement and community development in the age of algorithms. New Media & Society 21, 7 (2019), 1417–1443.

[26]

Leandro Silva, Mainack Mondal, Denzil Correa, Fabrício Benevenuto, and Ingmar Weber. 2016. Analyzing the Targets of Hate in Online Social Media.

[27]

K. Topal, M. Koyuturk, and G. Ozsoyoglu. 2016. Emotion -and area-driven topic shift analysis in social media discussions. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). 510–518.

[28]

Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex Machina: Personal Attacks Seen at Scale. In Proceedings of the 26th International Conference on World Wide Web (Perth, Australia) (WWW ’17). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1391–1399.

[29]

Sara K. Yeo, Leona Yi-Fan Su, Dietram A. Scheufele, Dominique Brossard, Michael A. Xenos, and Elizabeth A. Corley. 2019. The effect of comment moderation on perceived bias in science news. Information, Communication & Society 22, 1 (2019), 129–146.

[30]

Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. 2016. Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, 3485–3495.

Cited By

Morini VCitraro SSajno ESansoni MRiva GStella MRossetti G(2025)Online posting effects: Unveiling the non-linear journeys of users in depression communities on RedditComputers in Human Behavior Reports10.1016/j.chbr.2024.10054217(100542)Online publication date: Mar-2025
https://doi.org/10.1016/j.chbr.2024.100542
Tabassum MMackey ASchuett ALerner ABalzarotti DXu W(2024)Investigating moderation challenges to combating hate and harassmentProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3698903(37-54)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3698903
Vargo CHopp TAgarwal P(2024)Balancing Brand Safety and User Engagement in a Two-Sided Market: An Analysis of Content Monetization on RedditJournal of Current Issues & Research in Advertising10.1080/10641734.2023.230162145:2(242-256)Online publication date: 31-Jan-2024
https://doi.org/10.1080/10641734.2023.2301621
Show More Cited By

Index Terms

Investigating Toxicity Across Multiple Reddit Communities, Users, and Moderators

Index terms have been assigned to the content through auto-classification.

Recommendations

Statistical Modeling of Harassment against Reddit Moderators
WWW '20: Companion Proceedings of the Web Conference 2020

Despite the dedication that some volunteer moderators of online communities display when performing their moderation duties, they become targets of hate and harassment by other users. To understand what causes the change in moderator role from heroes to ...
"Welcome!": social and psychological predictors of volunteer socializers in online communities
CSCW '13: Proceedings of the 2013 conference on Computer supported cooperative work

Volunteer socializers are members of a community who voluntarily help newcomers become familiar with the popular practices and attitudes of the community. In this paper, we explore the social and psychological predictors of volunteer socializers on ...
Community Archetypes: An Empirical Framework for Guiding Research Methodologies to Reflect User Experiences of Sense of Virtual Community on Reddit
CSCW

Humans need a sense of community (SOC), and social media platforms afford opportunities to address this need by providing users with a sense of virtual community (SOVC). This paper explores SOVC on Reddit and is motivated by two goals: (1) providing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '20: Companion Proceedings of the Web Conference 2020

April 2020

854 pages

ISBN:9781450370240

DOI:10.1145/3366424

Editors:
Amal El Fallah Seghrouchni
Sorbonne University, France
,
Gita Sukthankar
University of Central Florida, United States
,
Tie-Yan Liu
Microsoft Research Asia, China
,
Maarten van Steen
University of Twente, Netherlands

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 April 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '20

Sponsor:

SIGWEB

WWW '20: The Web Conference 2020

April 20 - 24, 2020

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
1,014
Total Downloads

Downloads (Last 12 months)143
Downloads (Last 6 weeks)16

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Morini VCitraro SSajno ESansoni MRiva GStella MRossetti G(2025)Online posting effects: Unveiling the non-linear journeys of users in depression communities on RedditComputers in Human Behavior Reports10.1016/j.chbr.2024.10054217(100542)Online publication date: Mar-2025
https://doi.org/10.1016/j.chbr.2024.100542
Tabassum MMackey ASchuett ALerner ABalzarotti DXu W(2024)Investigating moderation challenges to combating hate and harassmentProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3698903(37-54)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3698903
Vargo CHopp TAgarwal P(2024)Balancing Brand Safety and User Engagement in a Two-Sided Market: An Analysis of Content Monetization on RedditJournal of Current Issues & Research in Advertising10.1080/10641734.2023.230162145:2(242-256)Online publication date: 31-Jan-2024
https://doi.org/10.1080/10641734.2023.2301621
Corradini E(2023)The Dark Threads That Weave the Web of Shame: A Network Science-Inspired Analysis of Body Shaming on RedditInformation10.3390/info1408043614:8(436)Online publication date: 2-Aug-2023
https://doi.org/10.3390/info14080436
Guo QShi CYin ZLiu CMa X(2023)Exploring the Effects of Event-induced Sudden Influx of Newcomers to Online Pop Music Fandom Communities: Content, Interaction, and EngagementProceedings of the ACM on Human-Computer Interaction10.1145/36100637:CSCW2(1-24)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610063
Aji FIriani AHendry (2023)Toxicity and Sentiment Analysis About Digital Bounty on Social Media2023 12th International Conference on Awareness Science and Technology (iCAST)10.1109/iCAST57874.2023.10359318(289-294)Online publication date: 9-Nov-2023
https://doi.org/10.1109/iCAST57874.2023.10359318
Tariq NSyed ZSaba E(2023)Praise or Insult? Identifying Cyberbullying Using Natural Language Processing2023 7th International Multi-Topic ICT Conference (IMTIC)10.1109/IMTIC58887.2023.10178609(1-7)Online publication date: 10-May-2023
https://doi.org/10.1109/IMTIC58887.2023.10178609
Kokinda EMoster MDominic JRodeghero P(2023)Under the Bridge: Trolling and the Challenges of Recruiting Software Developers for Empirical Research Studies2023 IEEE/ACM 45th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)10.1109/ICSE-NIER58687.2023.00016(55-59)Online publication date: May-2023
https://doi.org/10.1109/ICSE-NIER58687.2023.00016
Spennemann D(2022)Persistence and Attrition among Participants in a Multi-Page Online Survey Recruited via Reddit’s Social Media NetworkSocial Sciences10.3390/socsci1102003111:2(31)Online publication date: 18-Jan-2022
https://doi.org/10.3390/socsci11020031
Bin Zia HRaman ACastro IHassan Anaobi IDe Cristofaro ESastry NTyson G(2022)Toxicity in the Decentralized Web and the Potential for Model SharingProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/35309016:2(1-25)Online publication date: 6-Jun-2022
https://dl.acm.org/doi/10.1145/3530901
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten