skip to main content
10.1145/2063576.2063675acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

A pairwise ranking based approach to learning with positive and unlabeled examples

Published: 24 October 2011 Publication History

Abstract

A large fraction of binary classification problems arising in web applications are of the type where the positive class is well defined and compact while the negative class comprises everything else in the distribution for which the classifier is developed; it is hard to represent and sample from such a broad negative class. Classifiers based only on positive and unlabeled examples reduce human annotation effort significantly by removing the burden of choosing a representative set of negative examples. Various methods have been proposed in the literature for building such classifiers. Of these, the state of the art methods are Biased SVM and Elkan & Noto's methods. While these methods often work well in practice, they are computationally expensive since hyperparameter tuning is very important, particularly when the size of labeled positive examples set is small and class imbalance is high. In this paper we propose a pairwise ranking based approach to learn from positive and unlabeled examples (LPU) and we give a theoretical justification for it. We present a pairwise RankSVM (RSVM) based method for our approach. The method is simple, efficient, and its hyperparameters are easy to tune. A detailed experimental study using several benchmark datasets shows that the proposed method gives competitive classification performance compared to the mentioned state of the art methods, while training 3-10 times faster. We also propose an efficient AUC based feature selection technique in the LPU setting and demonstrate its usefulness on the datasets. To get an idea of the goodness of the LPU methods we compare them against supervised learning (SL) methods that also make use of negative examples in training. SL methods give a slightly better performance than LPU methods when there is a rich set of negative examples; however, they are inferior when the number of negative training examples is not large enough.

References

[1]
C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In ICML '05: Proceedings of the Twenty Second International Conference on Machine Learning, 2005.
[2]
B. Calvo, P. Larranaga, and J. A. Lozano. Learning Bayesian classifiers from positive and unlabeled examples. Pattern Recognition Letters, 28:2375--2384, 2007.
[3]
B. Calvo, P. Larranaga, and J. A. Lozano. Feature subset selection from positive and unlabeled examples. Pattern Recognition Letters, 30:1027--1036, 2009.
[4]
C.-C. Chang and C.-J. Lin. LIBSVM: a library for support vector machines, 2001. Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm.
[5]
O. Chapelle. Software for Rank SVM. http://olivier.chapelle.cc/primal/ranksvm.m.
[6]
O. Chapelle and S. S. Keerthi. Multi-class feature selection with support vector machines. In Proceedings of the American Statistical Association, 2008.
[7]
O. Chapelle and S. S. Keerthi. Efficient algorithms for ranking with SVMs. To appear in Information Retrieval Journal, Special Issue on Learning to Rank, 2009.
[8]
F. Denis, R. Gilleron, and M. Tommasi. Text classification classifiers from positive and unlabeled examples. In The 9th International Conf. Information Processing and Management of Uncertainty in Knowledge Based Systems, pages 1927--1934, 2002.
[9]
C. Elkan and K. Noto. Learning classifiers from only positive and unlabeled data. In KDD '08: Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 213--220, New York, NY, USA, 2008. ACM.
[10]
G. Forman. An extensive empirical study of feature selection metrics for text classification. Journal of Machine Learning Research, pages 1289--1306, 2003.
[11]
Y. Freund, R. Iyer, R. Schapire, and Y. Singer. An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4:933--969, 2003.
[12]
R. Herbrich, T. Graepel, and K. Obermayer. Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers, pages 115--132. MIT Press.
[13]
T. Joachims. A support vector method for multivariate performance measures. In ICML '05: Processing of International conference on Machine learning, pages 377--384, New York, NY, USA, 2005. ACM.
[14]
T. Joachims. Training linear SVMs in linear time. In KDD '06: Processing of Knowledge Discovery and Data Mining. ACM, 2006. Software available at http://www.cs.cornell.edu/People/tj/svm_light/svm_rank.html.
[15]
K. Lang. Newsweeder: Learning to filter netnews. In ML '95: Proceedings of the 12th International Machine Learning Conference, pages 331--339, 1995.
[16]
W. S. Lee and B. Liu. Learning with positive and unlabeled examples using weighted logistic regression. In ICML '03: Proceedings of the Twentieth International Conference on Machine Learning, pages 448--455, 2003.
[17]
X. Li and B. Liu. Learning to classify texts using positive and unlabeled data. In Proceedings of International Joint Conferences on Artificial Intelligence, pages 587--594, 2003.
[18]
B. Liu, Y. Dai, X. Li, W. S. Lee, and P. S. Yu. Building text classifiers using positive and unlabeled examples. In ICDM '03: Proceedings of the Third IEEE International Conference on Data Mining, pages 179--186. IEEE Computer Society, 2003.
[19]
B. Liu, W. S. Lee, P. S. Yu, and X. Li. Partially supervised classification of text documents. In ICML '02: Proceedings of the Nineteenth International Conference on Machine Learning, pages 387--394, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc.
[20]
L. M. Manevitz, M. Yousef, N. Cristianini, J. Shawe-taylor, and B. Williamson. One-class SVMs for document classification. Journal of Machine Learning Research, 2:139--154, 2001.
[21]
V. Sindhwani and S. S. Keerthi. Large scale semi-supervised linear SVMs. Technical report, 2006. available at http://www.keerthis.com/semisup_techreport_06.ps.
[22]
X. Wu, R. K. Srihari, and Z. Zheng. Document representation for one-class SVM. In Proceedings of European Conference on Machine Learning, pages 489--500, 2004.
[23]
H. Yu, J. Han, and K. C.-C. Chang. PEBL: Web page classification without negative examples. IEEE Transactions on Knowledge and Data Engineering, 16:70--81, 2004.
[24]
H. Yu, C. Zhai, and J. Han. Text classification classifiers from positive and unlabeled documents. In Proc. 12th International Conf. Information and Knowledge Management, pages 232--239, 2003.
[25]
Z. Zheng, X. Wu, and R. Srihari. Feature selection for text categorization on imbalanced data. In ACM KDD Explorations Newsletter, pages 80--89, 2004.

Cited By

View all
  • (2024)Class-Imbalanced-Aware Distantly Supervised Named Entity RecognitionIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.325208435:9(12117-12129)Online publication date: Sep-2024
  • (2024)Revising the Problem of Partial Labels from the Perspective of CNNs' Robustness2024 IEEE/ACIS 22nd International Conference on Software Engineering Research, Management and Applications (SERA)10.1109/SERA61261.2024.10685603(88-93)Online publication date: 30-May-2024
  • (2024)Positive-Unlabelled learning for identifying new candidate Dietary Restriction-related genes among ageing-related genesComputers in Biology and Medicine10.1016/j.compbiomed.2024.108999180(108999)Online publication date: Sep-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge management
October 2011
2712 pages
ISBN:9781450307178
DOI:10.1145/2063576
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2011

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. classification
  2. learning with positive and unlabeled examples
  3. pairwise ranking
  4. support vector machines

Qualifiers

  • Research-article

Conference

CIKM '11
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)17
  • Downloads (Last 6 weeks)1
Reflects downloads up to 10 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Class-Imbalanced-Aware Distantly Supervised Named Entity RecognitionIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.325208435:9(12117-12129)Online publication date: Sep-2024
  • (2024)Revising the Problem of Partial Labels from the Perspective of CNNs' Robustness2024 IEEE/ACIS 22nd International Conference on Software Engineering Research, Management and Applications (SERA)10.1109/SERA61261.2024.10685603(88-93)Online publication date: 30-May-2024
  • (2024)Positive-Unlabelled learning for identifying new candidate Dietary Restriction-related genes among ageing-related genesComputers in Biology and Medicine10.1016/j.compbiomed.2024.108999180(108999)Online publication date: Sep-2024
  • (2022)An Effective Approach for Multi-label Classification with Missing Labels2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys)10.1109/HPCC-DSS-SmartCity-DependSys57074.2022.00259(1713-1720)Online publication date: Dec-2022
  • (2021)A ranking based multi-view learning method for positive and unlabeled graph classificationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2021.3119626(1-1)Online publication date: 2021
  • (2021)A new method for positive and unlabeled learning with privileged informationApplied Intelligence10.1007/s10489-021-02528-7Online publication date: 12-Jun-2021
  • (2021)A new dictionary-based positive and unlabeled learning methodApplied Intelligence10.1007/s10489-021-02344-zOnline publication date: 14-Apr-2021
  • (2020)Learning from positive and unlabeled data: a surveyMachine Learning10.1007/s10994-020-05877-5Online publication date: 2-Apr-2020
  • (2019)Positive And Unlabeled Learning Algorithms And Applications: A Survey2019 10th International Conference on Information, Intelligence, Systems and Applications (IISA)10.1109/IISA.2019.8900698(1-8)Online publication date: Jul-2019
  • (2019)TEDM-PU: A Tax Evasion Detection Method Based on Positive and Unlabeled Learning2019 IEEE International Conference on Big Data (Big Data)10.1109/BigData47090.2019.9006325(1681-1686)Online publication date: Dec-2019
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media