Global least-squares vs. EM training for the Gaussian mixture of experts

Abstract

Since the introduction of the mixture of experts models and the EM algorithm for training them, maximum likelihood training of such networks has been shown to be a very useful and powerful tool for function estimation and prediction. A similar architecture is derived by other researchers from the application of fuzzy rules. Such systems are often trained by a straightforward global error minimisation procedure. This paper argues that in certain situations global optimisation is the most appropriate approach to take despite its apparent lack of statistical justification compared to the maximum likelihood approach. Moreover a composition of the two approaches often gives the minimal error on both the training and validation sets.