Abstract
We present a novel adaptive feedforward neural network for online learning from doubly-streaming data, where both the data volume and feature space grow simultaneously. Traditional online learning and feature selection algorithms can’t handle this problem because they assume that the feature space of the data stream remains unchanged. We propose a Single Hidden Layer Feedforward Neural Network with Shortcut Connections (SLFN-S) that learns if a data stream needs to be mapped using a non-linear transformation or not, to speed up the learning convergence. We employ a growing strategy to adjust the model complexity to the continuously changing feature space. Finally, we use a weight-based pruning procedure to keep the run time complexity of the proposed model linear in the size of the input feature space, for efficient learning from data streams. Experiments with trapezoidal data streams on 8 UCI datasets were conducted to examine the performance of the proposed model. We show that SLFN-S outperforms the state of the art learning algorithm from trapezoidal data streams [16].
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Blondel, M., Kubo, Y., Naonori, U.: Online passive-aggressive algorithms for non-negative matrix factorization and completion. In: Artificial Intelligence and Statistics, pp. 96–104 (2014)
Crammer, K., Kulesza, A., Dredze, M.: Adaptive regularization of weight vectors. Mach. Learn. 91(2), 155–187 (2013)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Huang, G.B.: Learning capability and storage capacity of two-hidden-layer feedforward networks. IEEE Trans. Neural Netw. 14(2), 274–281 (2003)
Huang, G.B., Chen, Y.Q., Babri, H.A.: Classification ability of single hidden layer feedforward neural networks. IEEE Trans. Neural Netw. 11(3), 799–801 (2000)
Huang, G.B., Zhu, Q.Y., Siew, C.K.: Real-time learning capability of neural networks. IEEE Trans. Neural Netw. 17(4), 863–878 (2006)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Lee, K.M., Street, W.N.: An adaptive resource-allocating network for automated detection, segmentation, and classification of breast cancer nuclei topic area: image processing and recognition. IEEE Trans. Neural Netw. 14(3), 680–687 (2003)
Liang, N.Y., Huang, G.B., Saratchandran, P., Sundararajan, N.: A fast and accurate online sequential learning algorithm for feedforward networks. IEEE Trans. Neural Netw. 17(6), 1411–1423 (2006)
Raiko, T., Valpola, H., LeCun, Y.: Deep learning made easier by linear transformations in perceptrons. In: Artificial Intelligence and Statistics, pp. 924–932 (2012)
Schraudolph, N.N.: Centering neural network gradient factors. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 205–223. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_14
Szegedy, C., et al.: Going deeper with convolutions. In: CVPR (2015)
Vatanen, T., Raiko, T., Valpola, H., LeCun, Y.: Pushing stochastic gradient towards second-order methods – backpropagation learning with transformations in nonlinearities. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8226, pp. 442–449. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42054-2_55
Yingwei, L., Sundararajan, N., Saratchandran, P.: A sequential learning scheme for function approximation using minimal radial basis function neural networks. Neural Comput. 9(2), 461–478 (1997)
Zhai, K., Boyd-Graber, J.: Online latent dirichlet allocation with infinite vocabulary. In: International Conference on Machine Learning, pp. 561–569 (2013)
Zhang, Q., Zhang, P., Long, G., Ding, W., Zhang, C., Wu, X.: Online learning from trapezoidal data streams. IEEE Trans. Knowl. Data Eng. 28(10), 2709–2723 (2016)
Acknowledgments
This research is supported by the US National Science Foundation (NSF) under grants 1652107 and 1763620. The authors would like to thank Dr. Amirhossein Tavanaei for constructive criticism of the manuscript.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Beyazit, E., Hosseini, M., Maida, A., Wu, X. (2018). Learning Simplified Decision Boundaries from Trapezoidal Data Streams. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds) Artificial Neural Networks and Machine Learning – ICANN 2018. ICANN 2018. Lecture Notes in Computer Science(), vol 11139. Springer, Cham. https://doi.org/10.1007/978-3-030-01418-6_50
Download citation
DOI: https://doi.org/10.1007/978-3-030-01418-6_50
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-01417-9
Online ISBN: 978-3-030-01418-6
eBook Packages: Computer ScienceComputer Science (R0)