Abstract
The dual neural network-based (DNN) k-winner-take-all (kWTA) model is one of the simplest analog neural network models for the kWTA process. This paper analyzes the behaviors of the DNN-kWTA model under these two imperfections. The two imperfections are, (1) the activation function of IO neurons is a logistic function rather than an ideal step function, and (2) there are multiplicative Gaussian noise in the inputs. With the two imperfections, the model may not be able to perform correctly. Hence it is important to estimate the probability of the imperfection model performing correctly. We first derive the equivalent activation function of IO neurons under the two imperfections. Next, we derive the sufficient conditions for the imperfect model to operate correctly. These results enable us to efficiently estimate the probability of the imperfect model generating correct outputs. Additionally, we derive a bound on the probability that the imperfect model generates the desired outcomes for inputs with a uniform distribution. Finally, we discuss how to generalize our findings to handle non-Gaussian multiplicative input noise.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Touretzky, S.: Winner-take-all networks of \( O (n) \) complexity. Advances in Neural Information Processing Systems, (1) Morgan Kaufmann, 703–711 (1989)
Kwon, T.M., Zervakis, M.: KWTA networks and their applications. Multidimensional Systems and Signal Processing 6(4), 333–346 (1995)
Narkiewicz J.D., Burleson W.P.: Rank-order filtering algorithms: A comparison of VLSI implementations. In the 1993 IEEE International Symposium on Circuits and Systems. IEEE, 1941–1944 (1993)
Sum, J.P., Leung, C.S., Tam, P.K., Young, G.H., Kan, W.K., Chan, L.w.: Analysis for a class of winner-take-all model. IEEE transactions on neural networks 10(1), 64–71 (1999)
Hu, X., Wang, J.: An improved dual neural network for solving a class of quadratic programming problems and its \(k\)-winners-take-all application. IEEE Transactions on Neural networks 19(12), 2022–2031 (2008)
Moscovici, A.: High speed A/D converters: understanding data converters through SPICE, vol. 601. Springer Science & Business Media (2001)
Feng, R., Leung, C.S., Sum, J., Xiao, Y.: Properties and performance of imperfect dual neural network-based \(k\)WTA networks. IEEE transactions on neural networks and learning systems 26(9), 2188–2193 (2014)
Redouté, J.M., Steyaert, M.: Measurement of emi induced input offset voltage of an operational amplifier. Electronics Letters 43(20), 1088–1090 (2007)
Kuang, X., Wang, T., Fan, F.: The design of low noise chopper operational amplifier with inverter. In: 2015 IEEE 16th International Conference on Communication Technology (ICCT). pp. 568–571. IEEE (2015)
Lee, P.: Low noise amplifier selection guide for optimal noise performance. Analog Devices Application Note, AN-940 (2009)
Feng, R., Leung, C.S., Sum, J.: Robustness analysis on dual neural network-based \(k\)WTA with input noise. IEEE transactions on neural networks and learning systems 29(4), 1082–1094 (2017)
Sum, J., Leung, C.S., Ho, K.I.J.: On Wang \(k\)WTA with input noise, output node stochastic, and recurrent state noise. IEEE transactions on neural networks and learning systems 29(9), 4212–4222 (2017)
Semenova, N., et al.: Fundamental aspects of noise in analog-hardware neural networks. Chaos: An Interdisciplinary Journal of Nonlinear Science 29(10) (2019)
Kariyappa, S., et al.: Noise-resilient DNN: Tolerating noise in PCM-based AI accelerators via noise-aware training. IEEE Transactions on Electron Devices 68(9), 4356–4362 (2021)
Haley, D.C.: Estimation of the dosage mortality relationship when the dose is subject to error. STANFORD UNIV CA APPLIED MATHEMATICS AND STATISTICS LABS, Tech. rep. (1952)
Radev, S.T., Mertens, U.K., Voss, A., Ardizzone, L., Kothe, U.: Bayesflow: Learning complex stochastic models with invertible neural network. IEEE Transactions on Neural Networks and Learning Systems 33(4), 1452–1466 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Lu, W., Leung, CS., Sum, J. (2023). Effect of Logistic Activation Function and Multiplicative Input Noise on DNN-kWTA Model. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds) Neural Information Processing. ICONIP 2022. Communications in Computer and Information Science, vol 1791. Springer, Singapore. https://doi.org/10.1007/978-981-99-1639-9_17
Download citation
DOI: https://doi.org/10.1007/978-981-99-1639-9_17
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-1638-2
Online ISBN: 978-981-99-1639-9
eBook Packages: Computer ScienceComputer Science (R0)