Tutorial: Supervised Learning for Prevalence Estimation

Abstract

Quantification is the task of estimating, given a set $\sigma$ of unlabelled items and a set of classes $\mathcal {C}$, the relative frequency (or “prevalence”) $p(c_{i})$ of each class $c_{i}\in \mathcal {C}$. Quantification is important in many disciplines (such as e.g., market research, political science, the social sciences, and epidemiology) which usually deal with aggregate (as opposed to individual) data. In these contexts, classifying individual unlabelled instances is usually not a primary goal, while estimating the prevalence of the classes of interest in the data is. Quantification may in principle be solved via classification, i.e., by classifying each item in $\sigma$ and counting, for all $c_{i}\in \mathcal {C}$, how many such items have been labelled with $c_{i}$. However, it has been shown in a multitude of works that this “classify and count” (CC) method yields suboptimal quantification accuracy, one of the reasons being that most classifiers are optimized for classification accuracy, and not for quantification accuracy. As a result, quantification has come to be no longer considered a mere byproduct of classification, and has evolved as a task of its own, devoted to designing methods and algorithms that deliver better prevalence estimates than CC. The goal of this tutorial is to introduce the main supervised learning techniques that have been proposed for solving quantification, the metrics used to evaluate them, and the most promising directions for further research.