A Challenging Dataset for Bias Detection: The Case of the Crisis in the Ukraine

Abstract

The use of disinformation and purposefully biased reportage to sway public opinion has become a serious concern. We present a new dataset related to the Ukrainian Crisis of 2014–2015 which can be used by other researchers to train, test, and compare bias detection algorithms. The dataset comprises 4,538 articles in English related to the crisis from 227 news sources in 43 countries (including the Ukraine) comprising 1.7M words. We manually classified the bias of each article as either pro-Russian, pro-Western, or Neutral, and also aligned each article with a master timeline of 17 major events. When trained on the whole dataset a simple baseline SVM classifier using doc2vec embeddings as features achieves an $F_{1}$ score of 0.86. This performance is deceptively high, however, because (1) the model is almost completely unable to correctly classify articles published in the Ukraine (0.07 $F_{1}$), and (2) the model performs nearly as well when trained on unrelated geopolitics articles written by the same publishers and tested on the dataset. As has been pointed out by other researchers, these results suggest that models of this type are learning journalistic styles rather than actually modeling bias. This implies that more sophisticated approaches will be necessary for true bias detection and classification, and this dataset can serve as an incisive test of new approaches.