Positive feedback loops lead to concept drift in machine learning systems

Abstract

We have derived conditions when unintended feedback loops occur in supervised machine learning systems. In this paper, we study an important problem of discovering and measuring hidden feedback loops. Such feedback loops occur in web search, recommender systems, healthcare, predictive public policing and other systems. As a possible cause of echo chambers and filter bubbles, these feedback loops tend to produce concept drifts in user behavior. We study systems in their context of use, because both learning algorithms and user interactions are important. Then we decompose the automation bias from the use of the system into users adherence to predictions and their usage rate to derive conditions for a feedback loop to occur. We also provide estimates for the size of a concept drift caused by the loop. A series of controlled simulation experiments with real-world and synthetic data support our findings. This paper builds on our prior results and elaborates the analytical model of feedback loops, extends the experiments, and provides practical application guidelines.

Appendices

Apprendix A: Proofs of existence conditions and size of effect

1.1 A.1 Proof of Proposition 1

Proof

As per definition of the positive feedback loop we need to show that loss decreases when we do not update the predictive model.

Let us consider sample mean loss $L_{1} = \frac{1}{|\text {G}_1 |} \sum _{(x_1, y_1) \in \text {G}_1}$ $l(y^1; f(x^1; \theta (\text {D}_0))$ on the test set $\text {G}_1$ after a single iteration of T and write down a recurrence relation for L:

$$\begin{aligned} L_{1} = L(z_{1}, f(x_{1}; \theta (\text {D}_0)) = (1-p) L_0 + p V_{1} \end{aligned}$$

where $V_{1} = s_0 + s_1 L_0$ is the expected loss for the user accepted item, and $L_0$ is the initial loss on $\text {D}_0$.

As the loss function is additive and bounded, the learning algorithm is symmetric and $\beta$-stable (3) with constant A then for the loop to exist we need at the first step $k = 1$

$$\begin{aligned} L_0 - L_1 = p(1 - s_1) L_0, \end{aligned}$$

which on average should be larger than the maximum expected change in the loss due to imperfect algorithm learning stability constrained by the constant A:

$$\begin{aligned} \mathbb {E\,}_{\text {D}_0} (L_0 - L_1)> A \cdot \mathbb {E\,}_{\text {D}_0} (L_1 + L_0) / 2 > A \, L_0. \end{aligned}$$

The expectation is taken over starting training sets $\text {D}_0 \subset X$.

Dividing by $L_0 > 0$, and transforming we get

$$\begin{aligned} p > p_0 = \frac{A}{1 - s_1}. \end{aligned}$$

$\square$

1.2 A.2 Proof of Proposition 2

Proof

If there is a feedback loop $p \ge p_0$ (4) then according to the Conjecture 1 [4], there exists a steady-state at which the feedback loop no longer proceeds, the loss stays the same $L_{inf} \ge 0$.

The composition of the test set $G_{inf}$ at steady state remains so that a fraction p of items are user decisions and the rest are unaffected $G_0$. Therefore, the expected loss at this state is

$$\begin{aligned} L_{inf} = (1 - p) L_0 + p \, V_{inf}, \end{aligned}$$

where $L_0$ is the expected loss on $G_0$. Taking $V_{inf} = s_0 + s_1 L_{inf}$, and $s_0 = 0$ we get

$$\begin{aligned} L_{inf} = L_0 \frac{1 - p}{1 - s_1 p}. \end{aligned}$$

Therefore, the size of the feedback loop is

$$\begin{aligned} L_0 - L_{inf} = L_0 \frac{p (1 - s_1)}{1 - s_1 p}. \end{aligned}$$

Note that if $p \ge p_0 = A / (1 - s_1)$ then $L_0 \ge L_{inf}$ if $0 \le s_1 \le 1 / (1 + A)$. It can be shown by induction backwards in time from the step just before steady-state that for any preceding round r the expected loss $L_r \ge L_{inf}$. $\square$

Fig. 8

Example of a positive feedback loop (left) for fixed model $f_0$, updated model $f_r$ on updated data $G_r$ and original data $G_0$. No feedback loop (right). Dataset: housing prices data. Metric: $R^2$. Model: GBR, steps before retraining $m = 20$, $n=151$, $A \approx 0.22$

Fig. 9

Example of a positive feedback loop (left) for fixed model $f_0$, updated model $f_r$ on updated data $G_r$ and original data $G_0$. No feedback loop (right). Dataset: synthetic data. Metric: $R^2$. Model: GBR, steps before retraining $m = 20$, $n=600$, $A \approx 0.09$

Appendix B: Examples with $R^2$ metric