An Empirical Evaluation of Probability Estimation with Neural Networks

Recent interest in neural networks by researchers across a wide spectrum of disciplines has provided convincing evidence of their ability to address classification problems. In this article, we consider the issue of evaluating the predictive capability of neural networks when the output values are to be treated as probabilities. We propose the use of a variant of a chi-square statistic, based on the Hosmer–Lemeshow statistic from logistic regression, to measure the goodness-of-fit of neural network models for two-group membership problems. Through experimentation with a large real-world database, we demonstrate the application of this statistic, and examine the effects of varying the number of nodes in the hidden layer on its value. Our empirical results suggest that this statistic can be very useful in identifying significant differences in the probability estimation accuracy of neural network models.