Abstract
Machine learning algorithms based on semi-supervised strategies have drawn the attention of researchers due to their ability to work with limited labeled data making use of huge number of unlabeled samples. Graph based semi-supervised algorithms make an assumption of similarity of examples in lower dimensional manifold and use an objective that ensures similarity of labels as enforced by the similarity graph. Such methods typically make use of a L2 regularization term to avoid over-fitting. Regularization term further ensures convexity of the overall objective leading to efficient learning algorithms. Addressing the problem of low-supervision and high class imbalance, prior work has shown state-of-the-art results for anomaly detection and other important classification problems by using a convex-concave objective. The current work analyses such performance improvements of convex-concave objective thoroughly. Our study indicates that a KL-Divergence based loss function for semi-supervised learning has performed much better than the convex-concave objective based on L2-Loss. It is also seen that the one-versus-rest setting for multi-class classification using convex-concave objective is performing much weaker compared to the naturally multi-class KL-Divergence based multi-class classification setting.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
The source code used to build k-NN graph is a sub module of the frame work available at: http://download.joachims.org/sgt_light/current/sgt_light.tar.gz.
References
Belkin, M., Matveeva, I., Niyogi, P.: Regularization and semi-supervised learning on large graphs. In: Shawe-Taylor, J., Singer, Y. (eds.) COLT 2004. LNCS (LNAI), vol. 3120, pp. 624–638. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-27819-1_43
Belkin, M., Niyogi, P.: Semi-supervised learning on Riemannian manifolds. Mach. Learn. 56(1–3), 209–239 (2004)
Belkin, M., Niyogi, P., Sindhwani, V.: Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. J. Mach. Learn. Res. 7(Nov), 2399–2434 (2006)
Belkin, M., Niyogi, P., Sindhwani, V.: On manifold regularization. In: AISTATS, p. 1 (2005)
Bengio, Y., Delalleau, O., Le Roux, N.: 11 Label Propagation and Quadratic Criterion, pp. 193–216. Semi-supervised Learning Edition. MIT Press, January 2006
Blum, A., Chawla, S.: Learning from labeled and unlabeled data using graph mincuts. In: Proceedings of the Eighteenth International Conference on Machine Learning, ICML 2001, pp. 19–26. Morgan Kaufmann Publishers Inc., San Francisco (2001). http://dl.acm.org/citation.cfm?id=645530.757779
Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training. In: Proceedings of the Eleventh Annual Conference on Computational Learning Theory, pp. 92–100. ACM (1998)
Cai, D., He, X., Han, J., Huang, T.S.: Graph regularized nonnegative matrix factorization for data representation. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1548–1560 (2010)
Chapelle, O., Schölkopf, B., Zien, A.: Label propagation and quadratic criterion (2006)
Chapelle, O., Scholkopf, B., Zien, A.: Semi-supervised learning (Chapelle, o. et al., eds.; 2006) [book reviews]. IEEE Trans. Neural Netw. 20(3), 542 (2009)
Druck, G., Mann, G., McCallum, A.: Learning from labeled features using generalized expectation criteria. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2008, pp. 595–602. ACM, New York (2008). https://doi.org/10.1145/1390334.1390436
Joachims, T.: Transductive learning via spectral graph partitioning. In: Proceedings of the 20th International Conference on Machine Learning (ICML-2003), pp. 290–297 (2003)
Liu, W., He, J., Chang, S.F.: Large graph construction for scalable semi-supervised learning. In: Proceedings of the 27th International Conference on Machine Learning (ICML-2010), pp. 679–686 (2010)
Nigam, K., McCallum, A.K., Thrun, S., Mitchell, T.: Text classification from labeled and unlabeled documents using EM. Mach. Learn. 39(2–3), 103–134 (2000). https://doi.org/10.1023/A:1007692713085
Pimplikar, R., Garg, D., Bharani, D., Parija, G.: Learning to propagate rare labels. In: Proceedings of the 23rd ACM International Conference on Information and Knowledge Management, CIKM 2014, pp. 201–210. ACM, New York (2014). https://doi.org/10.1145/2661829.2661982
Rifkin, R., Klautau, A.: In defense of one-vs-all classification. J. Mach. Learn. Res. 5(Jan), 101–141 (2004)
Scudder, H.: Probability of error of some adaptive pattern-recognition machines. IEEE Trans. Inf. Theor. 11(3), 363–371 (1965)
Subramanya, A., Bilmes, J.: Semi-supervised learning with measure propagation. J. Mach. Learn. Res. 12, 3311–3370 (2011). https://dl.acm.org/citation.cfm?id=1953048.2078212
Subramanya, A., Bilmes, J.: Semi-supervised learning with measure propagation. J. Mach. Learn. Res. 12(null), 3311–3370 (2011)
Szummer, M., Jaakkola, T.: Partially labeled classification with Markov random walks. In: Advances in Neural Information Processing Systems, pp. 945–952 (2002)
Vapnik, V.N.: An overview of statistical learning theory. IEEE Trans. Neural Netw. 10(5), 988–999 (1999)
Wang, J., Jebara, T., Chang, S.F.: Graph transduction via alternating minimization. In: Proceedings of the 25th International Conference on Machine learning, pp. 1144–1151. ACM (2008)
Yang, Z., Cohen, W.W., Salakhutdinov, R.: Revisiting semi-supervised learning with graph embeddings. arXiv preprint arXiv:1603.08861 (2016)
Yuille, A.L., Rangarajan, A.: The concave-convex procedure. Neural Comput. 15(4), 915–936 (2003). https://doi.org/10.1162/08997660360581958
Zhou, D., Bousquet, O., Lal, T.N., Weston, J., Schölkopf, B.: Learning with local and global consistency. In: Proceedings of the 16th International Conference on Neural Information Processing Systems, NIPS 2003, pp. 321–328. MIT Press, Cambridge (2003). http://dl.acm.org/citation.cfm?id=2981345.2981386
Zhou, D., Huang, J., Schölkopf, B.: Learning from labeled and unlabeled data on a directed graph. In: Proceedings of the 22nd International Conference on Machine learning, pp. 1036–1043. ACM (2005)
Zhou, D., Schölkopf, B.: A regularization framework for learning from graph data. In: ICML Workshop on Statistical Relational Learning and its Connections to Other Fields, vol. 15, pp. 67–68 (2004)
Zhu, X., Ghahramani, Z.: Learning from labeled and unlabeled data with label propagation. Technical Report (2002)
Zhu, X., Ghahramani, Z., Lafferty, J.: Semi-supervised learning using Gaussian fields and harmonic functions. In: Proceedings of the Twentieth International Conference on International Conference on Machine Learning, ICML 2003, pp. 912–919. AAAI Press (2003). http://dl.acm.org/citation.cfm?id=3041838.3041953
Zhu, X.J.: Semi-supervised learning literature survey. University of Wisconsin-Madison Department of Computer Sciences, Technical Report (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Sristy, N.B., Nunna, S.K., Somayajulu, D.V.L.N., Kumar, N.V.N. (2020). Convex vs Convex-Concave Objective for Rare Label Classification. In: B. R., P., Thenkanidiyoor, V., Prasath, R., Vanga, O. (eds) Mining Intelligence and Knowledge Exploration. MIKE 2019. Lecture Notes in Computer Science(), vol 11987. Springer, Cham. https://doi.org/10.1007/978-3-030-66187-8_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-66187-8_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-66186-1
Online ISBN: 978-3-030-66187-8
eBook Packages: Computer ScienceComputer Science (R0)