LEAF: A Less Expert Annotation Framework with Active Learning

Abstract

Many modern ML applications rely on large amounts of labeled data, which can be difficult and time-consuming to obtain. Active Learning (AL) is an advanced solution that addresses this problem. AL not only enables efficient training with limited data but also speeds up the labeling process and saves on labor costs. However, existing AL methods primarily focus on optimizing the query sampling strategy for single-task and fixed model scenarios, which is inefficient for real-world multi-task scenarios. In multi-task AL, multi-model hyperparameters optimization and multi-query strategies bring new challenges that require more labor. In this paper, we propose LEAF, a Less Expert Annotation Framework, to tackle those challenges and reduce the workload of both data experts and technical experts. In LEAF, we apply AutoML techniques to automatically optimize hyperparameters for multi-task and multi-model AL and design a heuristic adaptive query strategy for multi-query strategy in AL. Experimental results on three publicly available datasets show that our framework requires fewer iterations, less training time, and higher precision than conventional Active Learning frameworks. Additionally, we present a detailed case study that demonstrates the practical use and high quality of our proposed framework for real-world data annotation tasks.