Abstract
Support Vector Machines are a family of algorithms for the analysis of data based on convex Quadratic Programming. We derive randomized algorithms for training SVMs, based on a variation of Random Sampling Techniques; these have been successfully used for similar problems. We formally prove an upper bound on the expected running time which is quasilinear with respect to the number of data points and polynomial with respect to the other parameters, i.e., the number of attributes and the inverse of a chosen soft margin parameter. [This is the combined journal version of the conference papers (Balcázar, J.L. et al. in Proceedings of 12th International Conference on Algorithmic Learning Theory (ALT’01), pp. 119–134, [2001]; Balcázar, J.L. et al. in Proceedings of First IEEE International Conference on Data Mining (ICDM’01), pp. 43–50, [2001]; and Balcázar, J.L. et al. in Proceedings of SIAM Workshop in Discrete Mathematics and Data Mining, pp. 19–29, [2002]).]
Similar content being viewed by others
References
Adler, I., Shamir, R.: A randomized scheme for speeding up algorithms for linear and convex programming with high constraints-to-variable ratio. Math. Program. 61, 39–52 (1993)
Balcázar, J.L., Dai, Y., Watanabe, O.: A Random sampling technique for training support vector machines: for a primal-form maximal-margin classifiers. In: Proceedings of 12th International Conference on Algorithmic Learning Theory (ALT’01). Lecture Notes in Computer Science, vol. 2225, pp. 119–134. Springer, London (2001)
Balcázar, J.L., Dai, Y., Watanabe, O.: Provably fast training algorithms for support vector machines. In: Proceedings of First IEEE International Conference on Data Mining (ICDM’01), pp. 43–50. IEEE, Los Alamitos (2001)
Balcázar, J.L., Dai, Y., Watanabe, O.: Provably fast support vector regression using random sampling. In: Proceedings of SIAM Workshop in Discrete Mathematics and Data Mining, pp. 19–29. SIAM, Philadelphia (2002)
Bennett, K.P., Bredensteiner, E.J.: Duality and geometry in SVM classifiers. In: Proceedings of 17th International Conference on Machine Learning (ICML’2000), pp. 57–64. Morgan Kaufmann, San Mateo (2000)
Bennett, K.P., Campbell, C.: Support Vector Machines: Hype or Hallelujah? SIGKDD Explorations 2, 2 (2000)
Bennett, K.P., Mangasarian, O.L.: Robust linear programming discrimination of two linearly inseparable sets. Optim. Methods Softw. 1, 23–34 (1992)
Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Belmont (1995)
Bi, J., Bennett, K.P.: Duality, geometry, and support vector regression. In: Proceedings of Advances in Neural Information Processing Systems (NIPS’02), pp. 539–600. MIT Press, Cambridge (2002)
Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin classifiers. In: Proceedings of 5th Annual Conference on Computational Learning Theory (COLT’92), pp. 144–152. ACM, New York (1992)
Bradley, P.S., Mangasarian, O.L., Musicant, D.R.: Optimization methods in massive datasets. In: Abello, J., Pardalos, P.M., Resende, M.G.C. (eds.) Handbook of Massive Datasets, pp. 439–471. Kluwer Academic, Dordrecht (2002)
Cauwenberghs, G., Poggio, T.: Incremental and decremental support vector machine learning. In: Proceedings of Advances in Neural Information Processing Systems (NIPS’00), pp. 409–415. MIT Press, Cambridge (2000)
Clarkson, K.L.: A Las Vegas algorithm for linear programming when the dimension is small. In: Proceedings of 29th IEEE Symposium on Foundations of Computer Science (FOCS’88), pp. 452–456. IEEE, Los Alamitos (1988). (See [15] for a better version.)
Clarkson, K.L.: Las Vegas algorithms for linear and integer programming. J. ACM 42, 488–499 (1995)
Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20, 273–297 (1995)
Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines. Cambridge University Press, Cambridge (2000)
Cucker, F., Smale, S.: On the mathematical foundations of learning. Bull. Am. Math. Soc. 39(1), 1–49 (2001)
Evgeniou, T., Pontil, M., Poggio, T.: A unified framework for regularization networks and support vector machines. AI Memo No. 1654, AI Lab MIT (1999)
Fine, S., Scheinberg, K.: Efficient SVM training using low-rank kernel representations. J. Mach. Learn. Res. 2, 243–264 (2001)
Gärtner, B., Welzl, E.: A simple sampling lemma: Analysis and applications in geometric optimization. Discrete Comput. Geom. 25(4), 569–590 (2001)
Hush, D., Scovel, C.: Polynomial-time decomposition algorithms for support vector machines. Mach. Learn. 51, 51–71 (2003)
Keerthi, S.S., Gilbert, E.G.: Convergence of a generalized SMO algorithm for SVM classifier design. Mach. Learn. 46(1–3), 351–360 (2002)
Lee, Y., Mangasarian, O.L.: RSVM: Reduced Support Vector Machines. In: CD Proceedings of the SIAM International Conference on Data Mining, Chicago, 5–7 April 2001. SIAM, Philadelphia (2001). (Available from http://www.cs.wisc.edu/ olvi/olvi.html.)
Lin, C.J.: Linear convergence of a decomposition method for support vector machines. Technical Report. http://www.csie.ntu.edu.tw/~cjlin/papers/linearconv.pdf (2001)
Lin, C.J.: On the convergence of the decomposition method for support vector machines. IEEE Trans. Neural Networks 14, 1267–1281 (2002)
Martín, M.: On-line support vector machine regression. In: Proceedings of 13th European Conference on Machine Learning (ECML’02), pp. 282–294 (2002).
Osuna, E., Freund, R., Girosi, F.: An improved training algorithm for support vector machines. In: Proceedings of IEEE Workshop on Neural Networks for Signal Processing, pp. 276–285 (1997)
Pavlov, D., Mao, J., Dom, B.: Scaling up support vector machines using boosting algorithm. In: Proceedings of International Conference Pattern Rec., pp. 2219–2222 (2000)
Platt, J.: Fast training of support vector machines using sequential minimal optimization. In: Schölkopf, B., Burges, C.J.C., Smola, A.J. (eds.) Advances in Kernel Methods—Support Vector Learning, pp. 185–208. MIT Press, Cambridge (1999)
Schölkopf, B., Burges, C., Vapnik, V.: Extracting support data for a given task. In: Proceedings of First International Conference on Knowledge Discovery and Data Mining (KDD’95), pp. 252–257 (1995)
Schölkopf, B., Smola, A., Williamson, R.C.: A new support vector regression algorithm. In: Proceedings of Advances in Neural Information Processing Systems (NIPS’99), pp. 330–336. MIT Press, Cambridge (1999)
Schölkopf, B., Smola, A., Williamson, R.C., Bartlett, P.L.: New support vector algorithms. Neural Comput. 12, 1207–1245 (2000)
Smola, A.J., Schölkopf, B.: A tutorial on support vector regression. NeuroCOLT Technical Report NC-TR-98-030, Royal Holloway College, University of London (1998)
Author information
Authors and Affiliations
Corresponding author
Additional information
The first and the fourth authors started this research while visiting the Centre de Recerca Matemàtica of the Institute of Catalan Studies in Barcelona.
The first author was supported by IST Programme of the EU under contract number IST-1999-14186 (ALCOM-FT), Spanish Government TIC2004-07925-C03-02, and CIRIT 2001SGR-00252.
The second author conducted this research while she was with Department of Mathematical and Computing Sciences, Tokyo Institue of Technology, and was supported by a Grant-in-Aid (C-13650444) from Japanese Goverment.
The fourth author was supported in part by a Grant-in-Aid for Scientific Research on Priority Areas “Discovery Science” 1998–2000 from Japanese Goverment.
Rights and permissions
About this article
Cite this article
Balcázar, J.L., Dai, Y., Tanaka, J. et al. Provably Fast Training Algorithms for Support Vector Machines. Theory Comput Syst 42, 568–595 (2008). https://doi.org/10.1007/s00224-007-9094-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00224-007-9094-6