Abstract
Autoencoders are essential for training multi-hidden layer neural networks. Parametric autoencoder trainings often require user selections of hidden neuron numbers and kernel types. In this paper, a semi-parametric autoencoder training method based on self-organized learning and incremental learning is proposed. The cost function is constructed incrementally by nonparametric learning, and the model parameter is trained by parametric learning. First, a topology learning neural network such as growing neural gas or self-organizing incremental neural network is trained to obtain a discrete representation of the training data. Second, the correlations between different dimensions are modeled as a joint distribution by the neural network representation and kernel smoothers. Finally, the loss function is defined to be the regression prediction errors with each dimension as a response variable in density regression. The parameter of kernels is selected by gradient descent which minimizes the reconstruction error on a data subset. The proposed architecture has the advantage of high training space efficiency because of incremental training, and the advantage of automated selection of hidden neuron numbers. Experiments are carried out on 4 UCI datasets and an image interpolation task. Results show that the proposed methods outperform the perceptron architecture autoencoders and the restricted Boltzmann machine in the task of nonlinear feature learning.
Similar content being viewed by others
References
Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mane D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viegas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2016) TensorFlow: large-scale machine learning on heterogeneous distributed systems. ArXiv e-prints
Bache K, Lichman M (2013) UCI machine learning repository, 901:1. http://archive.ics.uci.edu/ml. Accessed 25 Mar 2018
Bodin E, Malik I, Ek CH, Campbell NDF (2017) Nonparametric inference for auto-encoding variational Bayes. ArXiv e-prints
Cherif A, Cardot H, Boné R (2011) SOM time series clustering and prediction with recurrent neural networks. Neurocomputing 74(11):1936–1944
Crammer K, Dekel O, Keshet J, Shalev-Shwartz S, Singer Y (2006) Online passive-aggressive algorithms. J Mach Learn Res 7:551–585
Csji BC (2001) Approximation with artificial neural networks. Ph.D. thesis, Faculty of Sciences, Etvs Lornd University
Druzhkov PN, Kustikova VD (2016) A survey of deep learning methods and software tools for image classification and object detection. Pattern Recognit Image Anal 26(1):9–15
Fischer A, Igel C (2012) An introduction to restricted Boltzmann machines. Springer, Berlin, pp 14–36. https://doi.org/10.1007/978-3-642-33275-3_2
Fritzke B et al (1995) A growing neural gas network learns topologies. Adv Neural Inf Process Syst 7:625–632
Furao S, Ogura T, Hasegawa O (2007) An enhanced self-organizing incremental neural network for online unsupervised learning. Neural Netw 20(8):893–903
Zhang H, Chow TW (2015) Organizing books and authors by multilayer SOM. IEEE Trans Neural Netw Learn Syst 27(12):2537
Kingma DP, Welling M (2013) Auto-encoding variational Bayes. ArXiv e-prints
Kohonen T (1998) The self-organizing map. Neurocomputing 21(1):1–6
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th international conference on neural information processing systems, vol 1, NIPS’12, pp 1097–1105. Curran Associates Inc., USA. http://dl.acm.org/citation.cfm?id=2999134.2999257. Accessed 25 Mar 2018
Nalisnick E, Smyth P (2017) Stick-breaking variational autoencoders. In: International conference on learning representations (ICLR). http://par.nsf.gov/biblio/10039928. Accessed 25 Mar 2018
Oliphant TE (2015) Guide to NumPy, 2nd edn. CreateSpace Independent Publishing Platform, Scotts Valley
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825–2830
Scott DW (2015) Multivariate density estimation: theory, practice, and visualization. Wiley, New York
Shen F, Yu H, Sakurai K, Hasegawa O (2011) An incremental online semi-supervised active learning algorithm based on self-organizing incremental neural network. Neural Comput Appl 20(7):1061–1074
da Silva IN, Hernane Spatti D, Andrade Flauzino R, Liboni LHB, dos Reis Alves SF (2017) Self-organizing Kohonen networks. Springer, Cham, pp 157–172. https://doi.org/10.1007/978-3-319-43162-8_8
Silva TC, Zhao L (2012) Stochastic competitive learning in complex networks. IEEE Trans Neural Netw Learn Syst 23(3):385–398
Silverman BW (1986) Density estimation for statistics and data analysis, vol 26. CRC Press, Boca Raton
Snoek J, Adams RP, Larochelle H (2012) Nonparametric guidance of autoencoder representations using label information. J Mach Learn Res 13(1):2567–2588
Tfekci P (2014) Prediction of full load electrical power output of a base load operated combined cycle power plant using machine learning methods. Int J Electr Power Energy Syst 60:126–140
Thompson JJ, Blair MR, Chen L, Henrey AJ (2013) Video game telemetry as a critical tool in the study of complex skill learning. PloS one 8(9):e75,129
Tieleman T (2008) Training restricted Boltzmann machines using approximations to the likelihood gradient. In: Proceedings of the 25th international conference on machine learning, ICML ’08, pp 1064–1071. ACM, New York, NY, USA. https://doi.org/10.1145/1390156.1390290
Tomczak JM (2016) Learning informative features from restricted Boltzmann machines. Neural Process Lett 44(3):735–750. https://doi.org/10.1007/s11063-015-9491-9
Tsanas A, Little MA, McSharry PE, Ramig LO (2010) Accurate telemonitoring of Parkinson’s disease progression by noninvasive speech tests. IEEE Trans Biomed Eng 57(4):884–893
Voegtlin T (2002) Recursive self-organizing maps. Neural Netw 15(8):979–991
Xiang Z, Xiao Z, Wang D, Georges HM (2016) Incremental semi-supervised kernel construction with self-organizing incremental neural network and application in intrusion detection. J Intell Fuzzy Syst 31(2):815–823
Xiang Z, Xiao Z, Wang D, Li X (2016) A Gaussian mixture framework for incremental nonparametric regression with topology learning neural networks. Neurocomputing 194:34–44. https://doi.org/10.1016/j.neucom.2016.02.008
Xiang Z, Xiao Z, Wang D, Xiao J (2017) Gaussian kernel smooth regression with topology learning neural networks and python implementation. Neurocomputing. https://doi.org/10.1016/j.neucom.2017.01.051
Xin M, Zhang H, Sun M, Yuan D (2016) Recurrent temporal sparse autoencoder for attention-based action recognition. In: 2016 International joint conference on neural networks (IJCNN), pp 456–463. https://doi.org/10.1109/IJCNN.2016.7727234
Yang H, Wang B, Lin S, Wipf D, Guo M, Guo B (2015) Unsupervised extraction of video highlights via robust recurrent auto-encoders. In: 2015 IEEE international conference on computer vision (ICCV), pp 4633–4641. https://doi.org/10.1109/ICCV.2015.526
Zhang H, Cao X, Ho JKL, Chow TWS (2017) Object-level video advertising: an optimization framework. IEEE Trans Ind Inform 13(2):520–531. https://doi.org/10.1109/TII.2016.2605629
Zhao W, Xu L, Bai J, Ji M, Runge T (2017) Sensor-based risk perception ability network design for drivers in snow and ice environmental freeway: a deep learning and rough sets approach. Soft Comput 2:1–10
Acknowledgements
This work was supported in part by Fundamental Research Program of Shenzhen (Project No. JCYJ20170413162458312) and National Natural Science Foundations of China (No. 61562047).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Rights and permissions
About this article
Cite this article
Xiang, Z., Deng, C., Xiang, X. et al. Semi-parametric training of autoencoders with Gaussian kernel smoothed topology learning neural networks. Neural Comput & Applic 32, 4933–4950 (2020). https://doi.org/10.1007/s00521-018-3897-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-018-3897-z