Abstract
Supervised learning methods are powerful techniques to learn a function from a given set of labeled data, the so-called training data. In this paper the support vector machines approach for regression is investigated under a theoretical point of view that makes use of convex analysis and Fenchel duality. Starting with the corresponding Tikhonov regularization problem, reformulated as a convex optimization problem, we introduce a conjugate dual problem to it and prove that, whenever strong duality holds, the function to be learned can be expressed via the optimal solutions of the dual problem. Corresponding dual problems are then derived for different loss functions. The theoretical results are applied by numerically solving the regression task for two data sets and the accuracy of the regression when choosing different loss functions is investigated.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aronszajn, N. (1950). Theory of reproducing kernels. Transactions of the American Mathematical Society, 686, 337–404.
Boţ, R. I. (2010). Lecture notes in economics and mathematical systems: Vol. 637. Conjugate duality in convex optimization. Berlin/Heidelberg: Springer.
Boţ, R. I., & Lorenz, N. (2011). Optimization problems in statistical learning: duality and optimality conditions. European Journal of Operational Research, 213(2), 395–404.
Chapelle, O., Vapnik, V. N., & Weston, J. (1999). Transductive inference for estimating values of functions.
Geiger, C., & Kanzow, C. (2002). Theorie und Numerik restringierter Optimierungsaufgaben. Berlin/Heidelberg/New York: Springer.
Joachims, T. (2002). Learning to classify text using support vector machines: methods, theory and algorithms. Boston/Dordrecht/London: Kluwer Academic.
Kim, K. (2003). Financial time series forecasting using support vector machines. Neurocomputing, 55(1–2), 307–319.
Noble, W. S. (2004). Support vector machine application in computational biology. In B. Schölkopf, K. Tsuda, & J.-P. Vert (Eds.), Kernel methods in computational biology (pp. 71–92). Cambridge: MIT Press.
Rockafellar, R. T. (1970). Convex analysis. Princeton: Princeton University Press.
Saunders, C., Gammerman, A., & Vovk, V. (1998). Ridge regression learning algorithm in dual variables. In J. W. Shavlik (Ed.), Proceedings of the 15th international conference on machine learning (pp. 515–521). San Mateo: Morgan Kaufmann.
Schölkopf, B., Bartlett, P., Smola, A., & Williamson, R. (1999). Shrinking the tube: a new support vector regression algorithm. In M. S. Kearns, S. A. Solla, & D. A. Cohn (Eds.), Advances in neural information processing systems (Vol. 11, pp. 330–336). Cambridge: MIT Press.
Shawe-Taylor, J., & Cristianini, N. (2004). Kernel methods for pattern analysis. Cambridge: Cambridge University Press.
Stitson, M. O., Gammerman, A., Vapnik, V., Vovk, V., Watkins, C., & Weston, J. (1997). Support vector regression with ANOVA decomposition kernels. In B. Schölkopf, C. J. C. Burges, & A. J. Smola (Eds.), Advances in kernel methods: support vector learning (pp. 285–292). Cambridge: MIT Press.
Tikhonov, A. N., & Arsenin, V. Y. (1977). Solutions of ill-posed problems. Washington: W.H. Winston.
Van Gestel, T., Baesens, B., Garcia, J., & Van Dijcke, P. (2003). A support vector machine approach to credit scoring. Bank en Financiewezen, 2, 73–82.
Vapnik, V. N. (1995). The nature of statistical learning theory. New York: Springer.
Wahba, G. (1990). Series in applied mathematics: Vol. 59. Spline models for observational data. Philadelphia: SIAM.
Zhong, Y., Zhou, C., Huang, L., Wang, Y., & Yang, B. (2009). Support vector regression for prediction of housing values. In Proceedings of the international conference on computational intelligence and security CIS’09 (pp. 61–65).
Acknowledgements
The authors are thankful to two anonymous reviewers for remarks which improved the quality of the paper. The research of R.I. Boţ was partially supported by DFG (German Research Foundation), projects BO 2516/4-1 and WA 922/1-3.
The research of A. Heinrich was supported by the European Union, the European Social Fund (ESF) and prudsys AG in Chemnitz.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Boţ, R.I., Heinrich, A. Regression tasks in machine learning via Fenchel duality. Ann Oper Res 222, 197–211 (2014). https://doi.org/10.1007/s10479-012-1304-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10479-012-1304-1