Skip to main content
SpringerLink
Log in

On numerical optimization theory of infinite kernel learning

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

In Machine Learning algorithms, one of the crucial issues is the representation of the data. As the given data source become heterogeneous and the data are large-scale, multiple kernel methods help to classify “nonlinear data”. Nevertheless, the finite combinations of kernels are limited up to a finite choice. In order to overcome this discrepancy, a novel method of  “infinite”  kernel combinations is proposed with the help of infinite and semi-infinite programming regarding all elements in kernel space. Looking at all infinitesimally fine convex combinations of the kernels from the infinite kernel set, the margin is maximized subject to an infinite number of constraints with a compact index set and an additional (Riemann–Stieltjes) integral constraint due to the combinations. After a parametrization in the space of probability measures, it becomes semi-infinite. We adapt well-known numerical methods to our infinite kernel learning model and analyze the existence of solutions and convergence for the given algorithms. We implement our new algorithm called “infinite” kernel learning (IKL) on heterogenous data sets by using exchange method and conceptual reduction method, which are well known numerical techniques from solve semi-infinite programming. The results show that our IKL approach improves the classifaction accuracy efficiently on heterogeneous data compared to classical one-kernel approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Anderson E.J., Nash P.: Linear Programming in Infinite-Dimensional Spaces. Wiley, New York (1987)

    Google Scholar 

  2. Apostol T.M.: Mathematical Analysis: A Modern Approach to Advanced Calculus. Addison Wesley, Reading (1974)

    Google Scholar 

  3. Argyriou, A., Hauser, R., Micchelli, C., Pontil, M.: A dc-programming algorithm for kernel selection. In: Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA (2006)

  4. Bach, F.R., Lanckriet, G.R.G.: Multiple kernel learning, conic duality, and the smo algorithm. In: Proceedings of the 21st International Conference on Machine Learning (2004)

  5. Chang, C.-C., Lin, C.-J.: Libsvm: a library for support vector machines. http://www.csie.ntu.edu.tw/cjlin/libsvm (2001)

  6. Hettich R., Jongen H.Th.: Semi-infinite programming: conditions of optimality and applications. In: Stoer, J. (eds) Optimization Techniques 2. Lecture notes in control and Information Science, Springer, Berlin, Heidelberg, New York (1978)

    Google Scholar 

  7. Hettich R., Kortanek O.: Semi-infinite programming: theory, methods and applications. SIAM Rev. 35 3, 380–429 (1993)

    Article  Google Scholar 

  8. Hettich R., Zencke P.: Numerische Methoden der Approximation und semi-infiniten Optimierung. Tuebner, Stuttgart (1982)

    Google Scholar 

  9. Jongen H.Th., Jonker P., Twilt F.: Nonlinear Optimization in Finite Dimensions—Morse Theory, Chebyshev Approximation, Transversality, Flows, Parametric Aspects. Springer, New York (2000)

    Google Scholar 

  10. Ozan Y.: Scientific discussion with Yıldıray Ozan. Department of Mathematics, METU, Ankara (2008)

    Google Scholar 

  11. Özöğür-Akyüz, S.: A Mathematical Contribution of Statistical Learning and Continuous Optimization Using Infinite and Semi-Infinite Programming, to Computational Statistics (submitted). PhD thesis, Middle East Technical University, Insitiute of Applied Mathematics, Department of Scientific Computing, February (2009)

  12. Özöğür-Akyüz, S., Weber, G.W.: Modelling of kernel machines by infinite and semi-infinite programming. In: Hakim, A.H., Vasant, P., Barsoum, N. (eds.) Proceedings of the Second Global Conference on Power Control and Optimization, AIP Conference Proceedings, Bali, Indonesia. Mathematical and Statistical Physics, pp. 306–313. ISBN 978-0-7354-0696-4 (August 2009), 1–3 June (2009)

  13. Özöğür-Akyüz, S., Weber, G.-W.: Learning with infinitely many kernels via infinite and semi-infinite programming. Accepted to Optimization Methods and Software

  14. Özöğür-Akyüz, S., Hussain, Z., Shawe-Taylor, J.: Prediction with the SVM using test point margins. In: R. Sharda, S.V.B (eds) Special issue on Data Mining of Journal of (German B) In Annals of Information Systems. Springer Book Series (2009) (in press)

  15. Özöğür-Akyüz S., Shawe-Taylor J., Weber G.-W., Ögel Z.B.: Pattern analysis for the prediction of eukoryatic pro-peptide cleavage sites. Discret. Appl. Math. 157(10), 2388–2394 (2009)

    Article  Google Scholar 

  16. Özöğür-Akyüz, S., Weber, G.-W.: Learning with infinitely many kernels via semi-infinite programming. In: Continuous Optimization and Knowledge Based Technologies, 20th EURO Mini conference, Lithunia, pp. 342–348 (2008)

  17. Pardalos, P.M., Hansen, P. (eds.): Data Mining andMathematical Programming, volume CRM 45. American Mathematical Society, Providence (2008)

    Google Scholar 

  18. Rückmann J.-J., Weber G.-W.: Semi-infinite optimization: Excisional stability of noncompact feasible sets. Sibir. Math. Z. 39, 129–145 (1998)

    Google Scholar 

  19. Seref, O., Kundakcioglu, O.E., Pardalos, P.M. (eds.): Data Mining, Systems Analysis, and Optimization in Biomedicine. Springer, New York (2007)

    Google Scholar 

  20. Shiryaev A.N.: Probability. Springer, New York (1995)

    Google Scholar 

  21. Sonnenburg S., Räetsch G., Schafer C., Schölkopf B.: Large scale multiple kernel learning. J. Mach. Learn. Res. 7, 1531–1565 (2006)

    Google Scholar 

  22. Still, G.: Semi-infinite programming: An introduction, preliminary version. Technical report, University of Twente Department of Applied Mathematics, Enschede, The Netherlands (2004)

  23. Vaz, A.I.F., Fernandes, E.M.G.P., Gomes, M.P.S.F.: Discretization methods for semi-infinite programming. Investigaç ăo Operacional 21(1), (2001)

  24. Weber G.-W.: Generalized Semi-Infinite Optimization and Related Topics, volume 29 of Research and Exposition in Mathematics. Heldermann, Berlin (2003)

    Google Scholar 

  25. Yershova, A., LaVelle, S.M.: Deterministic sampling methods for spheres and so(3). In: IEEE International Conference on Robotics and Automation (ICRA) (2004)

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics and Computer Science, Bahçeşehir University, Istanbul, Turkey

    S. Özöğür-Akyüz

  2. Institute of Applied Mathematics, Middle East Technical University, METU, Ankara, Turkey

    G.-W. Weber

  3. Faculty of Economics, Management Science and Law, University of Siegen, Siegen, Germany

    G.-W. Weber

Authors
  1. S. Özöğür-Akyüz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. G.-W. Weber
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. Özöğür-Akyüz.

Additional information

This study is partially done at Instiute of Applied Mathematics, Middle East Technical University, Ankara, Turkey and Faculty of Engineering and Natural Sciences, Sabancı University, Istanbul, Turkey.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Özöğür-Akyüz, S., Weber, GW. On numerical optimization theory of infinite kernel learning. J Glob Optim 48, 215–239 (2010). https://doi.org/10.1007/s10898-009-9488-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-009-9488-x

Keywords

Search

Navigation