Revisiting squared-error and cross-entropy functions for training neural network classifiers

Abstract

This paper investigates the efficacy of cross-entropy and square-error objective functions used in training feed-forward neural networks to estimate posterior probabilities. Previous research has found no appreciable difference between neural network classifiers trained using cross-entropy or squared-error. The approach employed here, though, shows cross-entropy has significant, practical advantages over squared-error.

Appendix

This appendix contains information concerning the parameters used in generating the simulated distributions of the illustration problems.

1.1 Trivariate normal (z₁)

Let μ _ij and $\sigma^{2}_{ij}$ be the mean and variance of normal variable i for group j. The mean and variance parameters for Group 1 are $(\mu_{11}, \sigma^{2}_{11}) = (\mu_{21}, \sigma^{2}_{21}) = (\mu_{31}, \sigma^{2}_{31}) = (10.0, 25.0).$ For Group 2, the parameters are $(\mu _{12}, \sigma^{2}_{12}) = (\mu_{22}, \sigma^{2}_{22}) = (\mu _{32}, \sigma^{2}_{32}) = (5.5, 25.0)$ and for Group 3 $(\mu_{13}, \sigma^{2}_{13}) = (\mu_{23}, \sigma^{2}_{23}) = (\mu_{33}, \sigma^{2}_{33}) = (7.5, 25.0).$ Let Σ₁, Σ₂, and Σ₃ be the variance-covariance matrices for Groups 1, 2, and 3, respectively. For this example, Σ₁ = Σ₂ = Σ₃ = Σ where

$$ \Sigma = {\left( {\begin{array}{*{20}c} {{25.0}} & {{7.5}} & {{22.5}} \\ {{7.5}} & {{25.0}} & {{15.0}} \\ {{22.5}} & {{15.0}} & {{25.0}} \\ \end{array} } \right)}. $$

1.2 Bivariate bernoulli (z₂)

Let z = (Z_1j, Z_2j) be the bivariate Bernoulli variables for group j where P(Z_1j = 1) = p_1j, P(Z_2j = 1) = p_2j, P(Z_3j = 1) = p_3j, and ρ _j is the correlation coefficient. For Group 1, p₁₁ = 0.8, p₂₁=0.7, and ρ₁=0.2. For Group 2, p₁₂ = 0.5, p₂₂=0.55, and ρ₂=0.4. For Group 3, p₁₃ = 0.425, p₂₃=0.4, and ρ₃=0.6.

1.3 Weibull (z₃, z₄)

For Weibull variable i, let α _ij be the shape parameter for group j, and β _ij be the scale parameter. For this example, the parameters for the first Weibull variable z₃ are α₁₁=4.0, β₁₁=1.0, α₁₂=1.5, β₁₂=1.0, and α₁₃=2.0, β₁₃=1.0 and For the second Weibull variable z₄, α₂₁=0.35, β₂₁=1.0, α₂₂=0.55, β₂₂=1.0, and α₂₃=0.6, β₂₃=1.0. Therefore, z₃ is a concave density function and z₄ is convex.

1.4 Binomial (z₅)

Let t=1,2,...,T be the number of Bernoulli random variables composing the binomial random variate, and p _j be the probability that the Bernoulli random variable for group j is 1. Then μ _j = Tp _j and $\sigma^{2}_{j}=Tp_{j} (1 - p_{j}).$ For this example, p₁=0.5, p₂=0.3, p₃=0.7 and T=10.