Mohammad Jamhuri
ABSTRACT
This paper proposes a new approach to optimizing cost function for binary logistic regression by the Gauss-Newton method. This method was applied to the backpropagation phase as a part of the training process to update the weighted coefficients. To show the performance of the approach, we used two data sets to train the logistic regression model for binary classification problems. Our experiment demonstrated that the proposed methods could perform better than gradient descent for both examples, as we expected. Furthermore, the performance of our approach is more advanced than the classical method, either in speed or accuracy.
- M. J. Zaki and W. Meira Jr, Data Mining and Machine Learning: Fundamental Concepts and Algorithms. Cambridge University Press, 2020.Google Scholar
Cross Ref
- G. Strang, Linear algebra and learning from data. Wellesley-Cambridge Press Cambridge, 2019.Google Scholar
- J. Chee and P. Toulis, “Convergence diagnostics for stochastic gradient descent with constant learning rate,” in International Conference on Artificial Intelligence and Statistics, 2018, pp. 1476–1485.Google Scholar
- I. Kulikovskikh, S. Prokhorov, T. Lipić, T. Legović, and T. Šmuc, “BioGD: Bio-inspired robust gradient descent,” PLoS One, vol. 14, no. 7, p. e0219004, 2019.Google ScholarCross Ref
- L. Bottou, “Stochastic gradient descent tricks,” in Neural networks: Tricks of the trade, Springer, 2012, pp. 421–436.Google ScholarCross Ref
- R. W. Siregar and M. Ramli, “Analysis local convergence of gauss-newton method,” in IOP Conference Series: Materials Science and Engineering, 2018, vol. 300, no. 1, p. 12044.Google ScholarCross Ref
- K. Lange, “Newton's Method,” in Optimization, Springer, 2004, pp. 155–173.Google ScholarCross Ref
- P. Chen, “Hessian matrix vs. Gauss–Newton hessian matrix,” SIAM J. Numer. Anal., vol. 49, no. 4, pp. 1417–1435, 2011.Google ScholarDigital Library
- A. Botev, H. Ritter, and D. Barber, “Practical gauss newton optimisation for deep learning,” in International Conference on Machine Learning, 2017, pp. 557–565.Google Scholar
- S. Ruder, “An overview of gradient descent optimization algorithms,” arXiv Prepr. arXiv1609.04747, 2016.Google Scholar
Index Terms
- Performance Improvement of Logistic Regression for Binary Classification by Gauss-Newton Method
Recommendations
Logistic regression diagnostics in ridge regression
The adverse effects of multicollinearity and unusual observations are seen in logistic regression and attention had been given in the literature to each of these problems separately. However, multicollinearity and unusual observations can arise ...
Regularized logistic regression without a penalty term
Research highlights EDAs can be used to find regularized logistic classifiers. It avoids the determination of the regularization term. EDA is not influenced by large number of covariates. Yields to significant better performance on AUC measure, compared ...
Comparing logistic regression, neural networks, c5.0 and m5′ classification techniques
MLDM'12: Proceedings of the 8th international conference on Machine Learning and Data Mining in Pattern RecognitionThe aim of the paper is to compare the prediction accuracies obtained using logistic regression, neural networks (NN), C5.0 and M5′ classification techniques on 4 freely available data sets. For this a feedforward neural network with a single hidden ...
Comments