skip to main content
research-article
Open access

Game On, Hate Off: A Study of Toxicity in Online Multiplayer Environments

Published: 08 August 2024 Publication History

Abstract

The advent of online spaces, particularly social media platforms and video games, has brought forth a significant challenge: the detection and mitigation of toxic and harmful speech. This issue is not only pervasive but also detrimental to the overall user experience. In this study, we leverage small language models to reliably detect toxicity, achieving an average precision of 0.95. Analyzing eight months of chat data from two Ubisoft games, we uncover patterns and trends in toxic behavior. The insights derived from our research will contribute to the development of healthier online communities and inform preventive measures against toxicity.

1 Introduction

Detecting toxic and harmful speech in online spaces poses a significant challenge, particularly in light of the escalating consumption and prevalence of toxic and hate speech across various social media platforms [8, 9, 20, 28] and within different video games [25]. As stated in the latest (2023) report from the Anti-Defamation League (ADL), toxicity in gaming is “now so pervasive that it has become the norm for many players” [3]. In ADL’s 2022 report, there was a pressing demand from the surveyed people for more decisive action and accountability from companies [2]. In response to that report, Representative Lori Trahan led the request towards the top gaming companies, Ubisoft included, on the proliferation of harassment and extremism found in online games [26]. The top companies all replied with their respective efforts around cultivating a healthier online community such as word censorship, (shadow) banning of users, or blocking them from communication [17, 19], as well as the usage of automated tools. As a result, in ADL’s 2023 (fifth annual) report, harassment of adults decreased for the first time in five years from 86% to 76%. However, they still found an increase in harassment from 67% in 2022 to 75% in teens and pre-teens. Outside of the legal issues for game companies, ADL also found that toxicity within games impacts the bottom line, as 20% of gamers spent less money due to the harassment they received.
While game companies are addressing the call to reduce toxicity and promote player health, a reliable toxicity detection method is necessary to be able to measure the success of their new preventative and regulatory measures. Therefore, the primary contribution of this article is the development of a reliable toxicity detection model that achieves an average precision score of 0.95. This model whose code is publicly available is particularly useful for researchers and practitioners interested in detecting, measuring, and understanding toxicity within online multiplayer gaming environments. Subsequently, we apply our model to eight months’ worth of in-game chat data, offering visual insights into toxicity trends within two games developed by Ubisoft. By providing a systematic method to identify potential causes of toxicity, our work facilitates the proposal, measurement, and implementation of preventive actions against such behaviors. The utility of this research extends to game developers, community managers, and policymakers who aim to foster healthier online communities and mitigate the negative impacts of toxic behavior. Ultimately, this study serves as a foundation for future research in creating more inclusive and enjoyable online gaming experiences.

2 Related Works

Toxic or harmful speech classification is a fundamental task in Natural Language Processing (NLP). Its evolution has mirrored broader trends in NLP methodologies, transitioning from traditional machine learning models with manual feature engineering [28] to the utilization of deep neural networks [10, 11, 12, 31]. More recently, the field has seen the adoption of pre-trained language models [5, 13, 15, 23], which have significantly improved performance across a range of NLP tasks.
Toxicity detection is not a solved problem, as it remains a persistent challenge, compounded by difficulties arising from out-of-distribution data, such as new vocabularies, and the lack of consensus regarding its precise definition [27]. To address this, our previous work [30] adopts categories defined in the disruptive behavior in online games published by the Fair Play Alliance (FPA). The FPA, comprising over 200 gaming companies, has a united mission to identify, define, and enable awesome player interactions, fair play, and healthy communities in online games by design.
Another one of the challenges in toxicity detection is the lack of contextual knowledge passed to the models. Previous studies, such as References [12] and [21], have experimented with including news article titles and usernames as additional context for comments on news articles. Similarly, Reference [10] incorporated Twitter user metadata for tweets. More recently, Reference [23] integrated the parent comment and discussion title of Wikipedia comments, suggesting that a broader context, beyond just the parent comment, may be necessary to consistently enhance performance. In our previous work [30], we extend these approaches by creating a setup where our model effectively uses chat history and metadata, leading to noticeable performance gains.
Research into perceived toxic behaviors in games has been well-documented through a range of studies and surveys [1, 2, 6, 32]. However, limited attention has been paid to toxicity trends within multiplayer games, with much of the focus centered on understanding trends specific to individual matches in the game, rather than examining changes over time. Notable examples include investigations into trends within games such as League of Legends [7, 14], DOTA 2 [29], and World of Tanks [24]. These studies provide valuable insights but often lack a longitudinal perspective.
In our study, we contribute by presenting an exploratory analysis of toxicity trends not only within two additional games but also over a span of time. This provides a broader perspective on the evolving landscape of toxicity in gaming. Studying trends is crucial, as it allows us to understand how toxicity evolves and adapts over time, which can inform the development of more effective mitigation strategies. Furthermore, understanding these trends can help us anticipate future challenges and adapt our models accordingly. By including literature on the effects of toxicity on players, especially longer-term effects, we aim to underscore the importance of our work and its potential to contribute to healthier gaming environments.

3 Methodology

This study employs a RoBERTa-base model, a transformer-based machine learning technique for NLP, pre-trained on unlabeled in-game chat data and fine-tuned on labeled datasets specific to two Ubisoft games: For Honor and Rainbow Six Siege. These games were selected for this study due to their distinct gameplay styles and communication patterns, providing a diverse and representative sample of multiplayer gaming environments. For Honor is a melee combat game that emphasizes personal skill and precise control, while Rainbow Six Siege is a tactical shooter game that requires strategic team collaboration1. Both games feature in-game text chat as a primary communication channel, allowing players to coordinate strategies, celebrate victories, or, unfortunately, engage in toxic behavior. Matches in these games typically last between 10 to 25 minutes, and the amount of communication can vary widely, depending on the match’s progression and the players’ engagement. Rainbow Six Siege was also listed as one of the more toxic games in ADL’s 2022 report [2].

3.1 Toxicity Detection Model

Traditionally, toxicity detection has been approached as a classification task using various methods, including traditional ML models with manual feature engineering [28], deep neural networks [11, 31], and pre-trained language models [5, 13, 16].
Taking into account the performance gain due to transformer-based language models, we chose to use RoBERTa-base model, one of the smallest discriminative language models. It leverages an attention mechanism that learns contextual relations between words (or sub-words) in a text. In contrast to smaller and more traditional models that were often pre-trained on a small amount of static data, RoBERTa is dynamically pre-trained on a very large corpus of data and is therefore especially well-suited for understanding the nuances and context of language. Compared to larger transformer-based language models such as ChatGPT, RoBERTa is much smaller, meaning that compute requirements is light enough that it can run on consumer-grade GPUs. Hence, this model can support live real-time toxicity detection if needed without any strong investment for new compute infrastructure.
We initiated our work by obtaining the weights for a RoBERTa-base model [18] from HuggingFace. We then extended the pre-training of this model with the masked language modeling (MLM) task performed on unlabeled in-game chat data. The details of our pre-training dataset and process are elaborated in Section 3.2. Subsequently, we fine-tuned the model separately for each game using our labeled toxicity dataset, as described in Section 3.3.

3.2 Pre-training

To prepare the model for pre-training, we collected unlabeled in-game chat data from both games, capturing different snapshots in time. This is to ensure a comprehensive and diverse set of interactions for the model to learn from. The specific snapshots were selected to cover a wide range of gameplay scenarios and player behaviors, thereby enhancing the robustness and generalizability of our model. Specifically, we amassed approximately 11 million lines from For Honor and 18 million lines from Rainbow Six Siege, as summarized in Table 1. We focused on regions where English communication was prominent. For For Honor, we gathered data from three snapshots in August 2021, May 2022, and October to December 2023. For Rainbow Six Siege, data were collected from two snapshots in July 2021 and October to December 2023.
Table 1.
DatasetSnapshots# of Matches# of Chat Lines
For Honor1456,1302,052,588
2363,0491,670,204
31,787,0757,786,219
 2,606,25411,509,011
Rainbow Six Siege1442,56210,585,273
2196,4977,455,248
 639,05918,040,521
Table 1. Pre-training Dataset for the Toxicity Detection Model
Labeled data used for fine-tuning are sampled from Snapshot 1.
During pre-training, we largely adhered to the original parameters and training settings for RoBERTa, including randomly masking and predicting 15% of input tokens. Following the dataset preprocessing, we concatenated lines of chat, but only those that belonged in the same matches and separated them by the “[SEP]” token to fit the “max_token_size.” We employed Adam optimizer and a linear scheduler with a weight decay of 0.01, \(\beta _1\) of 0.8, and \(\beta _2\) of 0.98 and a warm-up ratio of 5% of the dataset. The pre-training process continued for a maximum of 40 epochs, with early stopping based on loss. In our case, the model trained with a learning rate of \(1e-3\) halted at the 13th epoch. After conducting a hyperparameter search among learning rates of \(1e-3\), \(1e-4\), \(1e-5\), and \(1e-6\) over five runs on 20% of the training data, we determined an optimal learning rate of \(1e-3\).

3.3 Fine-tuning

For our labeled dataset, we sampled in-game chat from Snapshot 1 of each game, with higher probability for matches with higher volumes of chat lines. We structured our toxic categories in alignment with the “Disruptive Behavior in Online Game” framework established by the Fair Play Alliance [4]. These categories encompass Hate and Harassment, Threats, Minor Endangerment, Extremism, Scams and Ads, Insults and Flaming, Spam, and Other Offensive Text. Detailed class definitions and annotation guidelines are available below.

3.3.1 Annotator Details.

For For Honor and Rainbow Six Siege, annotators were recruited from social media with representation in game experience and self-identification with marginalized groups taken into consideration. Each annotator had to be at least 18 years old, have advanced proficiency in English, reside in the North American time zone and be active in the respective game. We define active as having played the respective game within the last year for at least 16 hours in the player versus player (PVP) mode. After the initial recruitment, a pilot test was conducted to further filter annotators to those that understood the task and aligned themselves on the common definition of toxicity based on examples shown. A total of 25 annotators were recruited. Each annotator was instructed to highlight the minimum span of contiguous words in a chat line that falls under a toxic category. If a span of words can fall under more than one toxic category, then they were to use the most severe category. Each chat line was annotated by three annotators, with full visibility of all previous chat lines of the match available. We also did not show any game-related events nor whether the player was reported. A total of 99,371 lines, corresponding to 5,340 matches from For Honor, and 95,612 lines, corresponding to 1,392 matches from Rainbow Six Siege, were labeled.

3.3.2 Detailed Toxicity Definitions.

Our dataset does not include any audio or visual data, and therefore, categories such as cheating, abuse of play, and antisocial actions are not within the scope of this model. We present the severity of the toxic classes in descending order.
(1)
Hate and Harassment: Identity-based hate or harassment (e.g., racism, sexism, homophobia) or bullying/mobbing (e.g., a group of players bullying one or more players).
(2)
Threats: Threats of violence, physical safety to another player, employee or property, terrorism, or releasing a player’s real-world personal information (e.g., doxxing).
(3)
Minor Endangerment: Sexual and/or aggressive actions towards minors or attempts to get minors to perform sexual activities.
(4)
Extremism: Extremist views (e.g., white supremacy), attempts to groom or recruit for an extremist group or repeated sharing of political, religious, or social beliefs.
(5)
Scams and Ads: Fraud/scamming (e.g., including phishing, account stealing, bad trades, or theft), posting inappropriate links (e.g., malware, dangerous websites, advertising exploits), and advertising of websites, services, cheats, or rival products.
(6)
Insults and Flaming: Insults or attacks on another player or team (not based on player or team’s real or perceived identity).
(7)
Spam: Excessive sharing of the same or similar words, phrases, emojis, or sharing (e.g., “kdjfklsjafkldjkla”).
(8)
Other Offensive Texts: Any other message not covered in the above categories that is offensive and/or harms a player’s reasonable enjoyment of the game.

3.3.3 Toxicity Dataset.

In Table 2, the distribution of lines for each class is presented for both For Honor and Rainbow Six Siege datasets. From the raw labeled dataset provided by annotators, labels were collated using a majority vote approach. In cases where differing spans of toxic content were identified within a single line of text by the three annotators, the minimum span was selected among them, and the category with the highest number of votes was assigned. In instances of a tie, where all three annotators had differing opinions, the class representing the most severe form of toxicity was chosen. The inter-coder reliability score, represented by Fleiss \(\kappa\), is reported in the table, with For Honor at 0.47 and Rainbow Six Siege at 0.54. Fleiss \(\kappa\) scores falling between 0.41 and 0.60 are considered to indicate a moderate strength of agreement, according to the established criteria, which range from poor, fair, moderate, good, to very good.
Table 2.
CategoryClassFor HonorRainbow Six Siege
Severely ToxicHate and Harassment4,4535,482
Threats421618
HarmfulMinor Endangerment109625
Extremism173392
Scams and Ads53456
ToxicInsults and Flaming11,3298,824
Slightly ToxicSpam2,21011,127
Other Offensive2,0773,117
Not ToxicNon-toxic78,29264,937
Fleiss \(\kappa\)0.470.54
Table 2. Class Distribution for For Honor and Rainbow Six Siege
Classes are sorted by severity level in descending order.

3.3.4 Fine-tuning Details.

During fine-tuning, we partitioned the dataset into train (60%), validation (20%), and test (20%) subsets. We employed an AdamW optimizer and a linear learning rate scheduler with an epsilon of \(1e-8\), weight decay of 0.1, warm-up steps of 400, and a learning rate of \(1e-5\). The model received input in the form of “[Chat History] [SEP] [Chat Line].” Within the “Chat History,” we included as many previous chat lines as feasible within the max_token_size of 512, each separated by the [SEP] token. The model’s task was to categorize the toxicity of the current chat line.

4 Results

In this section, we present the performance of our toxicity detection model and compare it against Cleanspeak and Google’s Perspective API. The decision to compare our model with Cleanspeak was motivated by the desire to benchmark against industry standards, particularly the prevalent use of keyword and regex-based methods for moderation. Similarly, we chose to contrast our model with Google’s Perspective AI to gauge its performance against a well-established and widely used toxicity detection model in recent literature. It is noteworthy that both our model and Perspective AI are underpinned by transformer-based language models [16]. Subsequently, we visualize the toxicity trends observed in our dataset.

4.1 Toxicity Detection Model

4.1.1 Pre-training.

Table 3 displays the results of pre-training on the full dataset with varying learning rates. The optimal learning rate range for pre-training is between \(1e-3\) and \(1e-4\), as indicated by the best loss values. However, a lower pre-training loss does not necessarily translate to better fine-tuning performance. Hence, to select the best pre-trained model, we compared the F1-score on 20% of the Rainbow Six Siege dataset. The model trained with a learning rate of \(1e-3\) demonstrated the best F1-score across five different data splits, and we therefore chose it for full fine-tuning.
Table 3.
Pre-trainingFine-tuning
Learning RateLossEpochPrecisionRecallF1-score
1e-31.941382.44 ± 0.4582.52 ± 0.6482.43 ± 0.54
1e-42.20782.20 ± 0.0082.87 ± 0.0082.40 ± 0.00
1e-52.57781.96 ± 0.0081.96 ± 0.0081.89 ± 0.00
1e-62.901281.24 ± 0.3481.70 ± 0.3381.34 ± 0.35
Table 3. Pre-training RoBERTA

4.1.2 Performance.

Table 4 presents the performance of our model averaged over five different data splits for both For Honor and Rainbow Six Siege datasets. We juxtapose this performance against that of Cleanspeak and Google’s Perspective API (v1alpha1) [16]. Cleanspeak is a paid tool that has “premier profanity filter and moderation”1 based on user-defined keywords and regexes, a common method used for moderation. A line is identified as toxic if any keywords or regular expressions are matched. Perspective API leverages a character-level transformer-based language model trained on toxicity observed in social media, serving as a standard baseline. As noted by the Perspective API research team, a chat line is classified as toxic if its score is greater than or equal to 0.7. Our game-specific trained model notably surpasses this benchmark across precision, recall, and F1-score metrics.
Table 4.
 For HonorRainbow Six Siege
 PrecisionRecallF1-scorePrecisionRecallF1-score
Cleanspeak66.6229.1040.4865.9138.9248.93
Perspective API73.4837.9750.0775.1124.3836.81
Our Model81.12 ± 0.4681.51 ± 0.5981.11 ± 0.5782.95 ± 0.3183.56 ± 0.2783.56 ± 0.27
Table 4. Toxicity Detection Model Performance against Perspective API
It is worth noting that the majority of toxicity detection models are tailored towards social media and blogs rather than in-game chat data. Among the few public toxicity datasets focused on game chat data, such as Reference [29], we observe a structural disparity in task objectives. They employ a dual-level annotation approach, wherein token-level annotations are automated, resembling a keyword-based system akin to Cleanspeak. At the utterance-level, annotations are conducted by two gamers and two non-gamers, without specification regarding their experience with the underlying game, DOTA 2. Moreover, the utterance-level annotations merely differentiate between toxic (implicitly or explicitly) and non-toxic utterances, neglecting various categories of toxicity essential for moderation purposes.

4.2 Toxicity Trends

With the fine-tuned toxicity detection model, we run inference for both games. The time-frame and daily average number of matches and players are shown in Table 5. Note, our data only includes the matches and players who have at least one line of chat. Matches without any text communication and players who do not type in chat are not in this dataset.
Table 5.
 For HonorRainbow Six Siege
No. of Matches (with Chat)27k92k
No. of Players (who Chat)13k127k
No. of Lines per Match4.3115.39
No. of Players per Match1.354.12
Table 5. Daily Average Statistics for For Honor and Rainbow Six Siege
Data are sampled from January 1, 2023, to August 31, 2023.

4.2.1 Matches and Players.

Figure 1 provides visualizations of the daily distribution of matches and unique players for each game. While both distributions are similar and definitely correlated, some interesting observations emerge. We observed a notable trend in For Honor where the number of matches exceeds the number of players, suggesting that players frequently engage in multiple matches per day. This is particularly interesting, as it contrasts with Rainbow Six Siege, where the number of players surpasses the number of matches, indicating a higher volume of chat communication relative to the number of matches played.
Fig. 1.
Fig. 1. Normalized distribution of matches and unique players for For Honor and Rainbow Six Siege. For normalization, we divide the total by the same constant for the number of matches and players within each game.
This distinction is significant, as it reflects the differing gameplay dynamics of the two games: For Honor emphasizes individual combat and strategy over extended periods, while Rainbow Six Siege involves team-based play with multiple rounds, offering more opportunities for strategic communication during preparation times.
Understanding these dynamics is crucial, as any future moderation policy or toxicity mitigation strategy introduced can impact how players communicate and their overall enjoyment of the game. It is important to note that one game’s player communication dynamic cannot be taken as a baseline for other games, especially when we have the data to analyze and compare. Therefore, our study provides valuable insights that can guide the development of game-specific strategies for managing player communication and mitigating toxicity.

4.3 Toxicity

4.3.1 Percentage of Toxic Matches.

Figure 2 illustrates the percentage of toxic matches, where a match is considered toxic if at least one chat line is flagged by our detection model. For For Honor, the percentage of toxic matches remains relatively stable at around 32%, while for Rainbow Six Siege, it exhibits more volatility, with an approximate baseline of 50%. Notably, as the number of matches played on a given day increases, the overall percentage of toxicity tends to decrease in both games. This trend aligns with the expectation that most players who communicate do not engage in toxic behavior. Peaks in toxicity coincide with specific in-game events, such as increased loot and free event passes in For Honor and the higher baseline toxicity in Rainbow Six Siege, likely due to its larger player base and more frequent updates.
Fig. 2.
Fig. 2. Percentage of toxic matches for For Honor and Rainbow Six Siege. Inset plot shows the normalized distribution of matches in blue and toxic matches in red.

4.3.2 Average Toxicity per Match.

Figure 3 explores the average toxicity within a game. The average number of lines per match differs significantly between For Honor (4.3 lines per match) and Rainbow Six Siege (12.4 lines per match). A notable observation is that, with more lines in a match, the percentage of toxic lines tends to decrease. Interestingly, in For Honor, there is a slight positive trend in the percentage of toxic lines per match, suggesting that while the number of toxic matches remains stable, the toxicity within individual matches is increasing.
Fig. 3.
Fig. 3. Percentage of toxic lines per match for For Honor and Rainbow Six Siege. Inset plot shows the average number of lines (blue) and toxic lines (red) per match.

4.3.3 Channels.

Figure 4 visualizes toxicity by channel, distinguishing between team-specific and public chat. In both games, approximately 40% of chat is team-specific, while 60% is public. A higher percentage of toxic chat lines is observed in the public channel, accounting for around 80% of toxic lines in both games, a finding similar to Reference [22].
Fig. 4.
Fig. 4. Normalized distribution of toxicity by channel for For Honor and Rainbow Six Siege. Normalization is done by percentage of toxic lines belonging to each channel per day.

4.3.4 Composition.

Figure 5 presents the composition of toxicity within chat lines. In For Honor, toxicity levels for each category remain relatively stable over time. Notably, a slight increase in the percentage of “slightly toxic” lines aligns with the peaks observed in Figure 2(a). In contrast, for Rainbow Six Siege, there is a higher proportion of “slightly toxic” lines and a lower percentage of “toxic” or “severely toxic” lines. In both games, nearly 85% of chat lines are not toxic.
Fig. 5.
Fig. 5. Normalized categorical toxicity composition for For Honor and Rainbow Six Siege. Normalization is done independently for each day to show the percentage of toxic lines belonging to each category per day.
In Figure 5, we look at the toxicity composition of the chat lines. We observe that, in Figure 5(a), toxicity level for each category is relatively stable. Interestingly, the peak in Figure 2(a) and Figure 3(a) is when there is a slight increase in the daily percentage of slightly toxic lines. When we compare Figure 5(b), we see that, percentage-wise, we have more slightly toxic lines and less toxic or severely toxic lines. In both games, we see that almost 85% of lines are in fact not toxic.

5 Conclusion

Our study has established a robust toxicity detection model that outperforms existing benchmarks, providing a solid foundation for future explorations in online gaming environments. The insights gained from analyzing toxicity trends pave the way for developing targeted interventions aimed at fostering inclusive and enjoyable online communities.
Moving forward, we envision the creation of intervention strategies that are informed by our findings. These could include real-time toxicity alerts, educational programs for players, and collaborative efforts within gaming companies to integrate healthier communication practices. To assess the effectiveness of these interventions, we propose comprehensive behavioral studies that track changes in player interactions over time. Longitudinal studies would be particularly valuable in understanding the long-term impact of interventions on player behavior and game culture. By continuing to refine our detection methods and collaborating on intervention strategies, we can contribute significantly to reducing toxicity and enhancing the overall gaming experience. Our commitment to this endeavor remains steadfast, as we seek to understand and mitigate the triggers of toxic behavior.

6 Limitations

While our study offers valuable insights, it is not without limitations. First, our analysis is limited to For Honor and Rainbow Six Siege and exclusively focuses on the English-speaking region. We consider a match is considered toxic if at least one chat line is flagged by our detection model. This is on the harsher side, where false positives could occur. Toxicity, furthermore, represents a complex and ever-evolving concept, and the performance of our model is contingent upon the accuracy and currency of its training data, which can quickly become outdated. Additionally, our analysis highlights the influence of external factors, such as the conclusion of in-game events, which we have not explicitly addressed. Finally, the biases found within the data nor the model has not been addressed.

Acknowledgments

We wish to thank Ubisoft La Forge, Ubisoft Montreal User Research Lab, and Ubisoft Data Office for providing technical support and insightful comments on this work.

Footnote

A Ethics Section

The data used within this study are obtained from Ubisoft and are NOT publicly available. The chat data are collected under Ubisoft’s terms and conditions. The research has been approved through McGill’s Research Ethics Board (REB# 23-05-074) and done under academic contract. Data can be acquired through proper agreements with Ubisoft. Model code is publicly available on GitHub: https://github.com/ubisoft/ubisoft-laforge-toxbuster

References

[1]
ADL. 2021. Online hate and harassment: The American experience 2021. Anti-Defamation League (Mar. 2021). Retrieved from DOI:https://www.adl.org/online-hate-2021
[2]
ADL. 2022. Hate and harassment in online games. Anti-Defamation League (Dec. 2022). Retrieved from DOI:https://www.adl.org/sites/default/files/documents/2022-12/Hate-and-Harassment-in-Online-Games-120622-v2.pdf
[3]
ADL. 2023. Hate is no game: Hate and harassment in online games 2023. Anti-Defamation League (Feb. 2023). Retrieved from DOI:https://www.adl.org/resources/report/hate-no-game-hate-and-harassment-online-games-2023
[4]
Fair Play Alliance. 2020. Disruption and harms in online gaming framework. Fair Play Alliance (12 2020), 1–48. Retrieved from DOI:https://fairplayalliance.org/wp-content/uploads/2022/06/FPA-Being-Targeted-about-Content-Moderation.pdf
[5]
Hind Almerekhi, Haewoon Kwak, Joni Salminen, and Bernard J. Jansen. 2022. PROVOKE: Toxicity trigger detection in conversations from the top 100 subreddits. Data Inf. Manag. (2022), 100019. DOI:
[6]
Nicole A. Beres, Julian Frommel, Elizabeth Reid, Regan L. Mandryk, and Madison Klarkowski. 2021. Don’t you know that you’re toxic: Normalization of toxicity in online gaming. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI’21). Association for Computing Machinery, New York, NY. DOI:
[7]
Jeremy Blackburn and Haewoon Kwak. 2014. STFU NOOB! Predicting crowdsourced decisions on toxic behavior in online games. In Proceedings of the 23rd International Conference on World Wide Web. 877–888. DOI:
[8]
Tuba Ciftci, Liridona Gashi, René Hoffmann, David Bahr, Aylin Ilhan, and Kaja Fietkiewicz. 2017. Hate speech on Facebook. In Proceedings of the 4th European Conference on Social Media Research. 425–433.
[9]
Nicola Döring and M. Mohseni. 2020. Gendered hate speech in YouTube and younow comments: Results of two content analyses. Stud. Commun. Media 9 (03 2020), 62–88. DOI:
[10]
Elise Fehn Unsvåg and Björn Gambäck. 2018. The effects of user features on Twitter hate speech detection. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2’18). Association for Computational Linguistics, 75–85. DOI:
[11]
Björn Gambäck and Utpal Kumar Sikdar. 2017. Using convolutional neural networks to classify hate-speech. In Proceedings of the 1st Workshop on Abusive Language Online. Association for Computational Linguistics, 85–90. DOI:
[12]
Lei Gao and Ruihong Huang. 2017. Detecting online hate speech using context aware models. In Proceedings of the International Conference Recent Advances in Natural Language Processing (RANLP’17). INCOMA Ltd., 260–266. DOI:
[13]
Manan Jhaveri, Devanshu Ramaiya, and Harveen Singh Chadha. 2022. Toxicity detection for indic multilingual social media content. CoRR abs/2201.00598 (2022).
[14]
Haewoon Kwak, Jeremy Blackburn, and Seungyeop Han. 2015. Exploring cyberbullying and other toxic behavior in team competition online games. 22 (04 2015). DOI:
[15]
Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. 2022. A New Generation of Perspective API: Efficient Multilingual Character-level Transformers. DOI:
[16]
Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. 2022. A new generation of perspective API: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD’22). Association for Computing Machinery, New York, NY, 3197–3207. DOI:
[17]
Robert Lewington. 2021. Being “Targeted” about content moderation:. Fair Play Alliance (04 2021), 1–21. Retrieved from https://fairplayalliance.org/wp-content/uploads/2022/06/FPA-Being-Targeted-about-Content-Moderation.pdf
[18]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019).
[19]
Brendan Maher. 2016. Can a video game company tame toxic behaviour? Nat. News 531 (03 2016), 568. DOI:
[20]
Shruthi Mohan, Apala Guha, Michael Harris, Fred Popowich, Ashley Schuster, and Chris Priebe. 2017. The impact of toxic language on the health of Reddit communities. In Proceedings of the Canadian Conference on Artificial Intelligence. 51–56. DOI:
[21]
Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017. Abusive language detection on Arabic social media. In Proceedings of the 1st Workshop on Abusive Language Online. Association for Computational Linguistics, 52–56. DOI:
[22]
Marcus Märtens, Siqi Shen, Alexandru Iosup, and Fernando Kuipers. 2015. Toxicity detection in multiplayer online games. DOI:
[23]
John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020. Toxicity detection: Does context really matter? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 4296–4305. DOI:
[24]
Cuihua Shen, Qiusi Sun, Taeyoung Kim, Grace Wolff, Rabindra Ratan, and Dmitri Williams. 2020. Viral vitriol: Predictors and contagion of online toxicity in World of Tanks. Comput. Hum. Behav. 108 (2020), 106343. DOI:
[25]
Bruno Silva, Mirian Tavares, Filipa Cerol, Susana Silva, Paulo Alves, and Beatriz Isca. 2020. Playing against hate speech—How teens see hate speech in video games and online gaming communities. J. Digit. Media Interact. 3 (07 2020), 34–52. DOI:
[26]
Lori Trahan. 2022. Summary of responses from gaming companies. Anti-Defamation League (Dec. 2022). Retrieved from DOI:https://trahan.house.gov/uploadedfiles/summary_responses_to_letter_game_companies_online_harassment_extremism.pdf
[27]
Betty Van Aken, Julian Risch, Ralf Krestel, and Alexander Löser. 2018. Challenges for Toxic Comment Classification: An In-depth Error Analysis. DOI:
[28]
Hajime Watanabe, Mondher Bouazizi, and Tomoaki Ohtsuki. 2018. Hate speech on Twitter: A pragmatic approach to collect hateful and offensive expressions and perform hate speech detection. IEEE Access 6 (2018), 13825–13835. DOI:
[29]
Henry Weld, Guanghao Huang, Jean Lee, Tongshu Zhang, Kunze Wang, Xinghong Guo, Siqu Long, Josiah Poon, and Soyeon Caren Han. 2021. CONDA: A CONtextual Dual-Annotated dataset for in-game toxicity understanding and detection. arxiv:2106.06213 [cs.CL]
[30]
Zachary Yang, Nicolas Grenon-Godbout, and Reihaneh Rabbany. 2023. Towards detecting contextual real-time toxicity for in-game chat. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 9894–9906. DOI:
[31]
Haoti Zhong, Hao Li, Anna Cinzia Squicciarini, Sarah Michele Rajtmajer, Christopher Griffin, David J. Miller, and Cornelia Caragea. 2016. Content-driven detection of cyberbullying on the Instagram social network. In Proceedings of the International Joint Conference on Artificial Intelligence.
[32]
Zicheng Zhu, Renwen Zhang, and Yuren Qin. 2022. Toxicity and prosocial behaviors in massively multiplayer online games: The role of mutual dependence, power, and passion. J. Comput.-mediat. Commun. 27, 6 (09 2022), zmac017. DOI:

Cited By

View all
  • (2024)Ethical Games: Toward Evidence-Based Guidance for Safeguarding Players and DevelopersGames: Research and Practice10.1145/36852072:2(1-11)Online publication date: 29-Jul-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Games: Research and Practice
Games: Research and Practice  Volume 2, Issue 2
June 2024
149 pages
EISSN:2832-5516
DOI:10.1145/3613575
Issue’s Table of Contents
This work is licensed under a Creative Commons Attribution-ShareAlike International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 August 2024
Online AM: 28 June 2024
Accepted: 07 June 2024
Revised: 02 June 2024
Received: 26 September 2023
Published in GAMES Volume 2, Issue 2

Check for updates

Author Tags

  1. Toxicity
  2. online multiplayer games
  3. toxicity trends
  4. chat moderation

Qualifiers

  • Research-article

Funding Sources

  • Ubisoft
  • Canadian Institute for Advanced Research (CIFAR AI Chair Program)
  • Natural Sciences and Engineering Research Council of Canada (NSERC) Postgraduate Scholarship-Doctoral (PGS D) Award
  • Fonds de recherche du Québec - Nature et Technologies (FRQNT) Doctoral Award

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1,434
  • Downloads (Last 6 weeks)320
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Ethical Games: Toward Evidence-Based Guidance for Safeguarding Players and DevelopersGames: Research and Practice10.1145/36852072:2(1-11)Online publication date: 29-Jul-2024

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Full Access

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media