Skip to main content
Springer Nature Link
Log in

Recipe for Fast Large-Scale SVM Training: Polishing, Parallelism, and More RAM!

  • Conference paper
  • First Online:
Artificial Intelligence and Machine Learning (BNAIC/Benelearn 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1805))

Included in the following conference series:

  • 254 Accesses

Abstract

Support vector machines (SVMs) are a standard method in the machine learning toolbox, in particular for tabular data. Non-linear kernel SVMs often deliver highly accurate predictors, however, at the cost of long training times. That problem is aggravated by the exponential growth of data volumes over time. It was tackled in the past mainly by two types of techniques: approximate solvers, and parallel GPU implementations. In this work, we combine both approaches to design an extremely fast dual SVM solver. We fully exploit the capabilities of modern compute servers: many-core architectures, multiple high-end GPUs, and large random access memory. On such a machine, we train a large-margin classifier on the ImageNet data set in 24 min.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The angle brackets denote the inner product in the kernel-induced feature space. We drop the bias or offset term [22].

  2. 2.

    These are surely not be the largest data sets in existence, but they are definitely large by the standards of the SVM literature.

  3. 3.

    A Cholesky decomposition is an attractive alternative at first glance, but since kernel matrices can be ill-conditioned, it regularly runs into numerical problems by requiring strict positive definiteness.

  4. 4.

    While this proceeding may yield a slightly optimistic bias (since some basis vectors may stem from the validation set), it is perfectly suitable for parameter tuning (since all parameter settings profit in the same way), and offers a considerable computational advantage.

  5. 5.

    https://eigen.tuxfamily.org/.

  6. 6.

    https://www.openmp.org/.

  7. 7.

    https://developer.nvidia.com/.

  8. 8.

    https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/.

  9. 9.

    https://www.image-net.org/.

  10. 10.

    https://github.com/TGlas/LPD-SVM.

  11. 11.

    https://github.com/Xtra-Computing/thundersvm.

  12. 12.

    https://github.com/EigenPro.

  13. 13.

    https://github.com/djurikom/BudgetedSVM.

References

  1. Bottou, L., Lin, C.J.: Support vector machine solvers (2006)

    Google Scholar 

  2. Braun, M.L.: Accurate error bounds for the eigenvalues of the kernel matrix. J. Mach. Learn. Res. 7, 2303–2328 (2006)

    MathSciNet  MATH  Google Scholar 

  3. Byvatov, E., Schneider, G.: Support vector machine applications in bioinformatics. Appl. Bioinform. 2(2), 67–77 (2003)

    Google Scholar 

  4. Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 1–27 (2011)

    Article  Google Scholar 

  5. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)

    Article  MATH  Google Scholar 

  6. Dekel, O., Singer, Y.: Support Vector Machines on a Budget. MIT Press, Cambridge (2007)

    Book  Google Scholar 

  7. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255 (2009)

    Google Scholar 

  8. Doğan, Ü., Glasmachers, T., Igel, C.: A unified view on multi-class support vector classification. J. Mach. Learn. Res. 17(45), 1–32 (2016)

    MathSciNet  MATH  Google Scholar 

  9. Doğan, Ü., Glasmachers, T., Igel, C.: A unified view on multi-class support vector classification. J. Mach. Learn. Res. (JMLR) 17(45), 1–32 (2016)

    MathSciNet  MATH  Google Scholar 

  10. Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: LIBLINEAR: a library for large linear classification. J. Mach. Learn. Res. 9, 1871–1874 (2008)

    Google Scholar 

  11. Glasmachers, T.: Finite sum acceleration vs. adaptive learning rates for the training of kernel machines on a budget. In: NIPS Workshop on Optimization for Machine Learning (2016)

    Google Scholar 

  12. Kecman, V., Brooks, J.P.: Locally linear support vector machines and other local models. In: The 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1–6. IEEE (2010)

    Google Scholar 

  13. Lin, C.J.: On the convergence of the decomposition method for support vector machines. IEEE Trans. Neural Networks 12(6), 1288–1298 (2001)

    Article  Google Scholar 

  14. Lu, W.C., Ji, X.B., Li, M.J., Liu, L., Yue, B.H., Zhang, L.M.: Using support vector machine for materials design. Adv. Manuf. 1(2), 151–159 (2013)

    Article  Google Scholar 

  15. Ma, S., Belkin, M.: Diving into the shallows: a computational perspective on large-scale shallow learning. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  16. Ma, Y., Guo, G.: Support Vector Machines Applications, vol. 649. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-02300-7

    Book  Google Scholar 

  17. Osuna, E., Freund, R., Girosi, F.: An improved training algorithm of support vector machines. In: Neural Networks for Signal Processing VII, pp. 276–285 (1997)

    Google Scholar 

  18. Qaadan, S., Schüler, M., Glasmachers, T.: Dual SVM training on a budget. In: Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods. SCITEPRESS - Science and Technology Publications (2019)

    Google Scholar 

  19. Rahimi, A., Recht, B.: Random features for large-scale kernel machines. In: Advances in Neural Information Processing Systems, pp. 1177–1184 (2008)

    Google Scholar 

  20. Shalev-Shwartz, S., Singer, Y., Srebro, N.: Pegasos: primal estimated sub-gradient solver for SVM. In: Proceedings of the 24th International Conference on Machine Learning, pp. 807–814 (2007)

    Google Scholar 

  21. Steinwart, I.: Sparseness of support vector machines. J. Mach. Learn. Res. 4, 1071–1105 (2003)

    MathSciNet  MATH  Google Scholar 

  22. Steinwart, I., Hush, D., Scovel, C.: Training SVMs without offset. J. Machine Learn. Res. 12(Jan), 141–202 (2011)

    Google Scholar 

  23. Tsang, I.W., Kwok, J.T., Cheung, P.M., Cristianini, N.: Core vector machines: fast SVM training on very large data sets. J. Mach. Learn. Res. 6(4) (2005)

    Google Scholar 

  24. Wang, Z., Crammer, K., Vucetic, S.: Breaking the curse of kernelization: budgeted stochastic gradient descent for large-scale SVM training. J. Mach. Learn. Res. 13(1), 3103–3131 (2012)

    MathSciNet  MATH  Google Scholar 

  25. Wen, Z., Shi, J., Li, Q., He, B., Chen, J.: ThunderSVM: a fast SVM library on GPUs and CPUs. J. Mach. Learn. Res. 19(1), 797–801 (2018)

    MathSciNet  Google Scholar 

  26. Yang, T., Li, Y.F., Mahdavi, M., Jin, R., Zhou, Z.H.: Nyström method vs. random fourier features: a theoretical and empirical comparison. In: Advances in Neural Information Processing Systems, pp. 476–484 (2012)

    Google Scholar 

  27. Zhang, K., Lan, L., Wang, Z., Moerchen, F.: Scaling up kernel SVM on limited resources: a low-rank linearization approach. In: Artificial Intelligence and Statistics, pp. 1425–1434. PMLR (2012)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the Deutsche Forschungsgemeinschaft under grant number GL 839/7-1.

Author information

Authors and Affiliations

  1. Ruhr-University Bochum, Bochum, Germany

    Tobias Glasmachers

Authors
  1. Tobias Glasmachers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tobias Glasmachers .

Editor information

Editors and Affiliations

  1. University of Antwerp, Antwerp, Belgium

    Toon Calders

  2. KU Leuven, Leuven, Belgium

    Celine Vens

  3. Ghent University, Gent, Belgium

    Jefrey Lijffijt

  4. University of Antwerp, Antwerp, Belgium

    Bart Goethals

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Glasmachers, T. (2023). Recipe for Fast Large-Scale SVM Training: Polishing, Parallelism, and More RAM!. In: Calders, T., Vens, C., Lijffijt, J., Goethals, B. (eds) Artificial Intelligence and Machine Learning. BNAIC/Benelearn 2022. Communications in Computer and Information Science, vol 1805. Springer, Cham. https://doi.org/10.1007/978-3-031-39144-6_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-39144-6_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-39143-9

  • Online ISBN: 978-3-031-39144-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics