Skip to main content
SpringerLink
Log in

A Kernel Approach to Multi-Task Learning with Task-Specific Kernels

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Several kernel-based methods for multi-task learning have been proposed, which leverage relations among tasks as regularization to enhance the overall learning accuracies. These methods assume that the tasks share the same kernel, which could limit their applications because in practice different tasks may need different kernels. The main challenge of introducing multiple kernels into multiple tasks is that models from different reproducing kernel Hilbert spaces (RKHSs) are not comparable, making it difficult to exploit relations among tasks. This paper addresses the challenge by formalizing the problem in the square integrable space (SIS). Specially, it proposes a kernel-based method which makes use of a regularization term defined in SIS to represent task relations. We prove a new representer theorem for the proposed approach in SIS. We further derive a practical method for solving the learning problem and conduct consistency analysis of the method. We discuss the relationship between our method and an existing method. We also give an SVM (support vector machine)-based implementation of our method for multi-label classification. Experiments on an artificial example and two real-world datasets show that the proposed method performs better than the existing method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Caruana R (1997) Multitask learning. Machine Learning 28(1):41–75

    Article  MathSciNet  Google Scholar 

  2. Ben-David S, Schuller R. Exploiting task relatedness for multiple task learning. In Lecture Notes in Computer Science, Carbonell J, Siekmann J (eds.), Springer, 2003, pp.567–580.

  3. Evgeniou T, Micchelli C, Pontil M (2005) Learning multiple tasks with kernel methods. Journal of Machine Learning Research 6(April):615–637

    MathSciNet  MATH  Google Scholar 

  4. Kato T, Kashima H, Sugiyama M, Asai K. Multi-task learning via conic programming. In Proc. the 22nd Conf. Neural Information Processing System, Dec. 2008, pp.737–744.

  5. Evgeniou T, Pontil M. Regularized multi-task learning. In Proc. the 10th SIGKDD, Aug. 2004, pp.109–117.

  6. Micchelli C, Pontil M. Kernels for multi-task learning. In Proc. the 19th NIPS, Dec. 2005, pp.921–928.

  7. Ando RK, Zhang T (2005) A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research 6(Nov.):1817–1853

    MathSciNet  MATH  Google Scholar 

  8. Argyriou A, Evgeniou T, Pontil M. Multi-task feature learning. In Proc. NIPS, December 2007, pp.41–48.

  9. Schölkopf B, Smola A (2002) Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, Massachusetts, USA

    Google Scholar 

  10. Hofmann T, Schölkopf B, Smola A (2008) Kernel methods in machine learning. Annals of Statistics 36(3):1171–1220

    Article  MathSciNet  MATH  Google Scholar 

  11. Lanckriet G R, Cristianini N, Bartlett P, Ghaoui L, Jordan M. Learning the kernel matrix with semi-definite programming. In Proc. the 19th ICML, July 2002, pp.323–330.

  12. Bach F, Lanckriet G R, Jordan M. Multiple kernel learning, conic duality, and the SMO algorithm. In Proc. the 21st ICML, July 2004, Article No. 6.

  13. Tang L, Chen J, Ye J. On multiple kernel learning with multiple labels. In Proc. the 21st IJCAI, July 2009, pp.1255–1260.

  14. Ji S, Sun L, Jin R, Ye J. Multi-label multiple kernel learning. In Proc. the 22nd NIPS, 2008, pp.777–784

  15. Duan L, Tsang I, Xu D, Chua T. Domain adaptation from multiple sources via auxiliary classifiers. In Proc. the 26th ICML, June 2009, pp.289–296.

  16. Aronszajn N (1950) Theory of reproducting kernels. Transactions of the American Mathematical Society 68(3):337–404

    Article  MathSciNet  MATH  Google Scholar 

  17. Cucker F, Smale S (2002) On the mathematical foundations of learning. Bulletin of the American Mathematical Society 39(1):1–49

    Article  MathSciNet  MATH  Google Scholar 

  18. Renardy M, Rogers R (1993) An Introduction to Partial Differential Equations. Springer-Verlag, New York, USA

    MATH  Google Scholar 

  19. Elisseeff A, Weston J. Kernel methods for multi-labelled classification and categorical regression problems. In Proc. the 16th NIPS, December 2002, pp.681–688.

  20. Lewis D. Evaluating text categorization. In Proc. the Work-shop on Speech and Natural Language, Feb. 1991, pp.312–318.

  21. Lanckriet G R, Deng M, Cristianini N, Jordan M, Noble W. Kernel-based data fusion and its application to protein function prediction in yeast. In Proc. Pacific Symp. Biocomputing, January 2004, pp.300–311.

  22. Dieudonné J (1969) Foundations of Modern Analysis, 2nd edn. Academic Press, New York, USA

    MATH  Google Scholar 

  23. Belkin M, Niyogi P, Sindhwani V (2006) Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research 7(Nov.):2399–2434

    MathSciNet  MATH  Google Scholar 

  24. Bartlett P, Mendelson S (2002) Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research 3(Nov.):463–482

    MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. MOE-Microsoft Key Laboratory of Statistics and Information Technology, Department of Probability and Statistics School of Mathematical Sciences, Peking University, Beijing, 100871, China

    Wei Wu

  2. Beijing International Center for Mathematical Research, Peking University, Beijing, 100871, China

    Wei Wu

  3. Noah’s Ark Lab, Huawei, Hong Kong Science Park, Hong Kong, China

    Hang Li (Member, CCF, ACM, IEEE)

  4. Microsoft Research Asia, Beijing, 100080, China

    Yun-Hua Hu (Member, ACM)

  5. Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, 48824, U.S.A.

    Rong Jin (Member, IEEE)

Authors
  1. Wei Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yun-Hua Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rong Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Wu.

Additional information

A short version of the paper has been published in the Proceedings of AAAI 2011.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wu, W., Li, H., Hu, YH. et al. A Kernel Approach to Multi-Task Learning with Task-Specific Kernels. J. Comput. Sci. Technol. 27, 1289–1301 (2012). https://doi.org/10.1007/s11390-012-1305-1

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-012-1305-1

Keywords

Search

Navigation