Abstract
Support vector regression (SVR) is a widely used regression technique for its competent performance. However, non-linear SVR is time consuming for large-scale tasks due to the dimension curse of kernelization. Recently, a budgeted stochastic gradient descent (BSGD) method has been developed to train large-scale kernelized SVC. In this paper, we extend the BSGD method to non-linear regression tasks. According to the performance of different budget maintenance strategies, we combine the stochastic gradient descent (SGD) method with the merging strategy. Experimental results on real-world datasets show that the proposed kernelized SVR with BSGD can achieve competent accuracy, with good computational efficiency compared to some state-of-the-art algorithms.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Boser BE, Guyon IM, Vapnik VN (1992) A training algorithm for optimal margin classifiers. In: Proceedings of the fifth annual workshop on Computational learning theory, ACM, pp 144–152
Brown JD, Summers MF, Johnson BA (2015) Prediction of hydrogen and carbon chemical shifts from rna using database mining and support vector regression. J Biomol NMR 63(1):1–14
Chen J, Xue X, Ha M, Yu D, Ma L (2014) Support vector regression method for wind speed prediction incorporating probability prior knowledge. Math Probl Eng 2014(2014):1–10
Osuna E, Freund R, Girosi F (1997) Training support vector machines: an application to face detection. In: Computer Vision and Pattern Recognition, 1997. Proceedings, 1997 IEEE Computer Society Conference on 1997, pp 130–136
Ho CH, Lin CJ (2012) Large-scale linear support vector regression. J Mach Learn Res 13(1):3323–3348
Lin CJ, Weng RC, Keerthi SS (2007) Trust region newton method for large-scale logistic regression. J Mach Learn Res 9(2):561–568
Hsieh CJ, Chang KW, Lin CJ, Keerthi SS, Sundararajan S (2008) A dual coordinate descent method for large-scale linear SVM. In: ICML, pp 1369–1398
Xie X, Chen C, Chen Z (2015) Mini-batch quasi-newton optimization for large scale linear support vector regression. In: International Conference on Mechatronics, Materials, Chemistry and Computer Engineering
Wang Y, Ou G, Pang W, Huang L, Coghill GM (2016) e-distance weighted support vector regression. CoRR. arXiv:1607.06657
Wang Z, Crammer K, Vucetic S (2012) Breaking the curse of kernelization: budgeted stochastic gradient descent for large-scale SVM training. J Mach Learn Res 13(1):3103–3131
Zheng S (2015) A fast algorithm for training support vector regression via smoothed primal function minimization. Int J Mach Learn Cybern 6(1):1–12
Lu J, Hoi SC, Wang J, Zhao P, Liu Z-Y (2016) Large scale online kernel learning. J Mach Learn Res 17(47):1–43
Crammer K, Kandola J, Singer Y (2003) Online classification on a budget. Adv Neural Inf Process Syst 40(2):225–232
Cavallanti G, Cesa-Bianchi N, Gentile C (2006) Tracking the best hyperplane with a simple budget perceptron. Mach Learn 69(2–3):143–167
Dekel O, Shalev-Shwartz S, Singer Y (2008) The forgetron: a kernel-based perceptron on a budget. SIAM J Comput 37(5):1342–1372
Orabona F, Keshet J, Caputo B (2009) Bounded kernel-based online learning. J Mach Learn Res 10(6):2643–2666
Lvesque JC (2013) Ensembles of budgeted kernel support vector machines for parallel large scale learning. In: NIPS 2013 Workshop on Big Learning: Advances in Algorithms and Data Management
Smola AJ, Scholkopf B (2004) A tutorial on support vector regression. Stat Comput 14(3):199–222
Cristianini N, Scholkopf B (2002) Support vector machines and kernel methods: the new generation of learning machines. Ai Mag 23(3):31–41
Bordes A, Bottou L, Gallinari P (2009) Sgd-qn: careful quasi-newton stochastic gradient descent. J Mach Learn Res 10(3):1737–1754
Zhu ZA, Chen W, Wang G, Zhu C, Chen Z (2009) P-packSVM: Parallel Primal grAdient desCent Kernel SVM. In: IEEE 13th International Conference on Data Mining, pp 677–686
Kivinen J, Smola AJ, Williamson RC (2004) Online learning with kernels. IEEE Trans Signal Process 52:2165–2176
Shalev-Shwartz S, Singer Y, Srebro N, Cotter A (2007), Pegasos: primal estimated sub-gradient solver for SVM. In: Machine Learning, Proceedings of the Twenty-Fourth International Conference, pp 3–30
Chang CC, Lin CJ (2011) Libsvm: a library for support vector machines. ACM Trans Intell Syst Technol 2(3):389–396
Acknowledgements
This work is supported by the National Natural Science Foundation of China under Grant Nos. 61432011, U1435212, and 61105054. This article is also supported by Commission for Collaborating Research Program, Key Laboratory of Solar Activity, National Astronomical Observatories, Chinese Academy of Sciences with KLSA201403.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Xie, Z., Li, Y. Large-scale support vector regression with budgeted stochastic gradient descent. Int. J. Mach. Learn. & Cyber. 10, 1529–1541 (2019). https://doi.org/10.1007/s13042-018-0832-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13042-018-0832-7