Abstract
In most large-scale real-world pattern classification problems, there is always some explicit information besides given training data, namely prior knowledge, with which the training data are organized. In this paper, we proposed a framework for incorporating this kind of prior knowledge into the training of min-max modular (M3) classifier to improve learning performance. In order to evaluate the proposed method, we perform experiments on a large-scale Japanese patent classification problem and consider two kinds of prior knowledge included in patent documents: patent’s publishing date and the hierarchical structure of patent classification system. In the experiments, traditional support vector machine (SVM) and M3-SVM without prior knowledge are adopted as baseline classifiers. Experimental results demonstrate that the proposed method is superior to the baseline classifiers in terms of training cost and generalization accuracy. Moreover, M3-SVM with prior knowledge is found to be much more robust than traditional support vector machine to noisy dated patent samples, which is crucial for incremental learning.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Liu B, Li X L, Lee WS, Yu P S. Text classification by labeling words. AAAI, 2004
Wu X Y, Srihari R. Incorporating prior knowledge with weighted margin support vector machines. In: Proceedings of International Conference on Knowledge Discovery and Data Mining, 2004, 326–333
Schapire R E, Rochery M, Rabim M, Gupta N. Boosting with prior knowledge for call classification. IEEE Transactions on Speech and Audio Processing, 2005, 13, 174–181
Zhu J B, Chen W L. Improving text categorization using domain knowledge In: Proceedings of International Conference on Applications of Natural Language to Information Systems, 2005, 103–113
Dayanik A, Lewis D D, Madigan D, Menkov V, Genkin A. Constructing informative prior distributions from domain knowledge in text classification. In: Proceedings of ACM’s Special Interest Group on Information Retrieval, 2006
Lu B L, Ito M. Task decomposition based on class relations: a modular neural network architecture for pattern classification. Biological and Artificial Computation: From Neuroscience to Technology. Springer, LNCS, 1997, 1240: 330–339
Lu B L, Ito M. Task decomposition and module combination based on class relations: A modular neural network for pattern classification. IEEE Transactions on Neural Networks, 1999, 10: 1244–1256
Anand R, Mehrotra K G, Mohan C K, Ranka S. An improved algorithm for neural network classification of imbalanced training sets. IEEE Transaction on Neural Netwook, 1993, 4: 962–969
Lu B L, Wang K A, Utiyama M, Isahara H. A part-versus-part method for massively parallel training of support vector machines. In: Proceedings of International Joint Conference on Neural Networks, 2004, 735–740
Krier M, Zaccá F. Automatic categorization applications at the European patent office. World Patent Information. Elsevier, 2002, 24(3): 187–196
Larkey L. Some issues in the automatic classification of US patents. Learning for Text Categorization. Technical Report WS-98-05, 1998, 87–90
Larkey L. A patent search and classification system. In: Proceedings of the fourth ACM conference on Digital libraries, 1999, 179–187
Mase H, Tsuji H, Kinukawa H, Ishihara M. Automatic patents categorization and its evaluation. Transactions of Information Processing Society of Japan(IPSJ), 1998
Fall C J, Benzineb K. Literature survey: Issues to be considered in the automatic classification of patents. World Intellectual Property Organization, 2002, 29
Fall C J, Torcsvári A, Benzineb K, Karetka G. Automated categorization in the international patent classification. In: Proceedings of ACM’s Special Interest Group on Information Retrieval. New York: ACM Press, 2003, 37: 10–25
Fujii A, Iwayama M, Kando N. Test collections for patent retrieval and patent classification in the 5th NTCIR workshop. In: Proceedings of the 5th international conference on language resources and evaluation, 2004, 1643–1646
Fujii A, Iwayama M, Kando N. Introduction to the special issue on patent processing. Information Processing and Management, 2007, 1149–1153
Wen Y M, Lu B L, Zhao H. Equal clustering makes min-max modular support vector machine more efficient. In: Proceedings of International Conference on Neural Information Processing, 2005, 77–82
Lian H C, Lu B L, Takikawa E, Hosoi S. Gender recognition using a min-max modular support vector machine. In: Proceedings of International Conference on Natural Computation, 2005, 438–441
Yang Y M, Pedersen J O. A comparattive study on feature selection in text categorization. In: Proceedings of International Conference on Machine Learning, 1997, 187–196
Sebastiani F. Machine learning in automated text categorization. ACM Computing Surveys, 2002, 34: 1–47
Zhao H, Lu B L. A modular k-nearest neighbor classification method for massively parallel text categorization. In: Proceedings of First International Symposium on Computational and Information Science. Springer, LNCS, 2004, 3314: 867–872
Wu K, Lu B L, Uchiyama M, Isahara H. An empirical comparison of min-max-modular k-NN with different voting methods to large-scale text categorization. Soft Computing — A Fusion of Foundations, Methodologies and Applications, 2008, 12(7): 647–655
Joachims T. Making large-scale support vector machine learning practical. Advances in Kernel Methods: Support Vector Learning. Cambridge: MIT Press, 1998
Lewis D D, Yang Y, Rose T, Li F. RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 2004, 5: 361–397
Liu W, Xue G R, Yu Y, Zeng H J. Importance-based web page classification using cost-sensitive SVM. In: Proceedings of International Conference on Web-Age Information Management, 2005, 127–137
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lu, B., Wang, X. & Utiyama, M. Incorporating prior knowledge into learning by dividing training data. Front. Comput. Sci. China 3, 109–122 (2009). https://doi.org/10.1007/s11704-009-0013-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11704-009-0013-7