skip to main content
10.1145/1571941.1571946acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article

Refined experts: improving classification in large taxonomies

Published: 19 July 2009 Publication History

Abstract

While large-scale taxonomies--especially for web pages--have been in existence for some time, approaches to automatically classify documents into these taxonomies have met with limited success compared to the more general progress made in text classification. We argue that this stems from three causes: increasing sparsity of training data at deeper nodes in the taxonomy, error propagation where a mistake made high in the hierarchy cannot be recovered, and increasingly complex decision surfaces in higher nodes in the hierarchy. While prior research has focused on the first problem, we introduce methods that target the latter two problems--first by biasing the training distribution to reduce error propagation and second by propagating up "first-guess" expert information in a bottom-up manner before making a refined top down choice. Finally, we present an empirical study demonstrating that the suggested changes lead to 10--30% improvements in F1 scores versus an accepted competitive baseline, hierarchical SVMs.

References

[1]
P. N. Bennett, S. T. Dumais, and E. Horvitz. The combination of text classifiers using reliability indicators. Information Retrieval, 8(1):67--100, 2004.
[2]
C. M. Bishop and M. Svensén. Bayesian hierarchical mixtures of experts. In UAI '03, 2003.
[3]
L. Cai and T. Hofmann. Hierarchical document categorization with support vector machines. In CIKM '04, 2004.
[4]
N. Cesa-Bianchi, C. Gentile, and L. Zaniboni. Hierarchical classification: combining bayes with svm. In ICML '06, 2006.
[5]
N. Cesa-Bianchi, C. Gentile, and L. Zaniboni. Incremental algorithms for hierarchical classification. Journal of Machine Learning Research, 7:31--54, 2006.
[6]
O. Dekel, J. Keshet, and Y. Singer. Large margin hierarchical classification. In ICML '04, 2004.
[7]
S. Dumais, E. Cutrell, and H. Chen. Optimizing search by showing results in context. In CHI '01, 2001.
[8]
S. T. Dumais and H. Chen. Hierarchical classification of Web content. In SIGIR '00, 2000.
[9]
T. Joachims. Text categorization with support vector machines: Learning with many relevant features. In ECML '98, 1998.
[10]
M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural Computation, 6:181--214, 1994.
[11]
A. R. Klivans and A. A. Sherstov. Improved lower bounds for learning intersections of halfspaces. In COLT '06, 2006.
[12]
D. Koller and M. Sahami. Hierarchically classifying documents using very few words. In ICML '97, 1997.
[13]
D. D. Lewis, Y. Yang, T. G. Rose, and F. Li. RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361--397, 2004.
[14]
W. Li and A. McCallum. Pachinko allocation: Dag-structured mixture models of topic correlations. In ICML '06, 2006.
[15]
T. Liu, Y. Yang, H. Wan, H. Zeng, Z. Chen, and W. Ma. Support vector machines classification with a very large-scale taxonomy. SIGKDD Explorations, 7(1):36--43, 2005.
[16]
A. McCallum, R. Rosenfeld, T. Mitchell, and A. Y. Ng. Improving text classification by shrinkage in a hierarchy of classes. In ICML '98, 1998.
[17]
D. M. Mimno, W. Li, and A. McCallum. Mixtures of hierarchical topics with pachinko allocation. In ICML '07, 2007.
[18]
Netscape Communication Corporation. Open directory project. http://www.dmoz.org.
[19]
J. C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, 1999.
[20]
M. E. Ruiz and P. Srinivasan. Hierarchical neural networks for text categorization. In SIGIR '99, 1999.
[21]
S. Shalev-Shwartz, Y. Singer, and N. Srebro. Pegasos: Primal estimated sub-GrAdient solver for svm. In ICML '07, 2007.
[22]
A. Sun and E. Lim. Hierarchical text classification and evaluation. In ICDM '01, 2001.
[23]
C. J. van Rijsbergen. Information Retrieval. Butterworths, London, 1979.
[24]
G.-R. Xue, D. Xing, Q. Yang, and Y. Yu. Deep classification in large-scale text hierarchies. In SIGIR '08, 2008.
[25]
Y. Yang and X. Liu. A re-examination of text categorization methods. In SIGIR '99, 1999.
[26]
B. Zhang, H. Li, Y. Liu, L. Ji, W. Xi, W. Fan, Z. Chen, and W.-Y. Ma. Improving web search results using affinity graph. In SIGIR '05, 2005.

Cited By

View all
  • (2024)Partial label learning for automated classification of single-cell transcriptomic profilesPLOS Computational Biology10.1371/journal.pcbi.101200620:4(e1012006)Online publication date: 5-Apr-2024
  • (2024)Sparse Feature-Persistent Hierarchical ClassificationNAECON 2024 - IEEE National Aerospace and Electronics Conference10.1109/NAECON61878.2024.10670617(147-152)Online publication date: 15-Jul-2024
  • (2023)Multi-Label Classification of Chinese Rural Poverty Governance Texts Based on XLNet and Bi-LSTM Fused Hierarchical Attention MechanismApplied Sciences10.3390/app1313737713:13(7377)Online publication date: 21-Jun-2023
  • Show More Cited By

Index Terms

  1. Refined experts: improving classification in large taxonomies

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
      July 2009
      896 pages
      ISBN:9781605584836
      DOI:10.1145/1571941
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 19 July 2009

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. large-scale hierarchy
      2. text classification

      Qualifiers

      • Research-article

      Conference

      SIGIR '09
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 792 of 3,983 submissions, 20%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)19
      • Downloads (Last 6 weeks)3
      Reflects downloads up to 25 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Partial label learning for automated classification of single-cell transcriptomic profilesPLOS Computational Biology10.1371/journal.pcbi.101200620:4(e1012006)Online publication date: 5-Apr-2024
      • (2024)Sparse Feature-Persistent Hierarchical ClassificationNAECON 2024 - IEEE National Aerospace and Electronics Conference10.1109/NAECON61878.2024.10670617(147-152)Online publication date: 15-Jul-2024
      • (2023)Multi-Label Classification of Chinese Rural Poverty Governance Texts Based on XLNet and Bi-LSTM Fused Hierarchical Attention MechanismApplied Sciences10.3390/app1313737713:13(7377)Online publication date: 21-Jun-2023
      • (2023)HmcNet: A General Approach for Hierarchical Multi-Label ClassificationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.320751135:9(8713-8728)Online publication date: 1-Sep-2023
      • (2023)Exploring and exploiting hierarchical structures for large-scale classificationInternational Journal of Machine Learning and Cybernetics10.1007/s13042-023-02039-615:6(2427-2437)Online publication date: 22-Dec-2023
      • (2022)Hierarchical Multilabel Ship Classification in Remote Sensing Images Using Label Relation GraphsIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2021.311111760(1-13)Online publication date: 2022
      • (2021)Emotion Recognition Robust to Indoor Environmental Distortions and Non-targeted Emotions Using Out-of-distribution DetectionACM Transactions on Computing for Healthcare10.1145/34923003:2(1-22)Online publication date: 20-Dec-2021
      • (2021)Hierarchical Metadata-Aware Document Categorization under Weak SupervisionProceedings of the 14th ACM International Conference on Web Search and Data Mining10.1145/3437963.3441730(770-778)Online publication date: 8-Mar-2021
      • (2021)Consistency-aware Multi-modal Network for Hierarchical Multi-label Classification in Online Education System2021 IEEE International Conference on Big Knowledge (ICBK)10.1109/ICKG52313.2021.00063(1-8)Online publication date: Dec-2021
      • (2021)Handling imbalance in hierarchical classification problems using local classifiers approachesData Mining and Knowledge Discovery10.1007/s10618-021-00762-835:4(1564-1621)Online publication date: 1-Jul-2021
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media