Authors:
Thomas Seidler
1
;
2
and
Markus Abel
1
;
2
Affiliations:
1
Ambrosys GmbH, Potsdam, Germany
;
2
Institute for Physics and Astronomy, Postsdam University, Potsdam, Germany
Keyword(s):
Phase Transition, Thermodynamics, Statistical Mechanics, Machine Learning, Image Classification, MNIST.
Abstract:
An important question for machine learning model concerns the achievable quality or performance of a model with respect to given data. In other words, we want to answer the question how robust a model is with respect to perturbation of the data. From statistical mechanics, a standard way to ”corrupt” input data is a study that uses additive noise to perturb data. This, in turn, corresponds to typical situations in processing data from any sensor as measurement noise. Larger models will often perform better, because they are able to capture more variance of the data. However, if the information content cannot be retrieved due to too large data corruptions a large network cannot compensate noise effects and no performance is gained by scaling the network. Here we study systematically the said effect, we add diffusive noise of increasing strength on a logarithmic scale to some well-known datasets for classification. As a result, we observe a sharp transition in training and test accurac
y as a function of the noise strength. In addition, we study if the size of a network can counterbalance the described noise. The transition observed resembles a phase transition as described in the framework of statistical mechanics. We draw an analogy between systems in statistical mechanics and Machine Learning systems that suggests general upper bounds for certain types of problems, described as the tuple (data, model). This is a fundamental result that may have large impact on practical applications.
(More)