A Theoretical Analysis of Semi-supervised Learning

Abstract

We analyze the dynamical behaviors of semi-supervised learning in the framework of on-line learning by using the statistical-mechanical method. A student uses several correlated input vectors in each update. The student is given a desired output for only one input vector out of these correlated input vectors. In this model, we derive simultaneous differential equations with deterministic forms that describe the dynamical behaviors of order parameters using the self-averaging property in the thermodynamic limit. We treat the Hebbian and Perceptron learning rules. As a result, it is shown that using unlabeled data is effective in the early stages for both of the two learning rules. In addition, we show that the two learning rules have qualitatively different dynamical behaviors. Furthermore, we propose a new algorithm that improves the generalization performance by switching the number of input vectors used in an update as the time step proceeds.