Authors:
Fabian Hinder
;
Valerie Vaquet
;
Johannes Brinkrolf
and
Barbara Hammer
Affiliation:
CITEC, Bielefeld University, Inspiration 1, Bielefeld, Germany
Keyword(s):
Concept Drift, Stream Learning, Drift Detection, No Free Lunch.
Abstract:
The notion of concept drift refers to the phenomenon that the distribution generating the observed data changes over time. If drift is present, machine learning models can become inaccurate and need adjustment. Many technologies for learning with drift rely on the interleaved test-train error to detect drift and trigger model updates. This type of drift detection is also used for monitoring systems aiming to detect anomalies. In this work, we analyze the relationship between concept drift and change of loss on a theoretical level. We focus on the sensitivity, specificity, and localization of change points in drift detection, putting an emphasize on the detection of real concept drift. With this focus, we compare the supervised and unsupervised setups which are already studied in the literature. We show that, unlike the unsupervised case, there is no universal supervised drift detector and that the assumed correlation between model loss and concept drift is invalid. We support our the
oretical findings with empirical evidence for a combination of different models and data sets. We find that many state-of-the-art supervised drift detection methods suffer from insufficient sensitivity and specificity, and that unsupervised drift detection methods are a promising addition to existing supervised approaches.
(More)