Kuo-Wei Hsu
ABSTRACT
Adaptive Boosting, or AdaBoost , is a meta-learning algorithm that employs a classification algorithm as a base learner to train classification models and uses these models to perform collective classification. One of its main features is that iteratively it forces the base learner to work more on difficult samples. Usually it can achieve better overall classification performance, when compared to a single classification model trained by the classification algorithm used as the base learner. SVM, short for Support Vector Machine, is a learning algorithm that employs a kernel to project the original data space to a data space where a hyperplane that can linearly separate as many samples of classes as possible can be found. Because both are top algorithms, researchers have been exploring the use of AdaBoost with SVM. Unlike others simply using SVM with a single kernel as the base learner in AdaBoost, we propose an approach that uses SVM with multiple kernels as the base learners in a variant of AdaBoost. Its main feature is that it not only considers difficulties of samples but also classification performance of kernels, and accordingly it selects as well as switches between kernels in the boosting process. The experiment results show that we can obtain better classification performance by using the proposed approach.
- E Alfaro, N García, M Gámez, and D Elizondo. 2008. Bankruptcy forecasting: An empirical comparison of AdaBoost and neural networks. Decis. Support Syst. 45, 1 (April 2008), 110--122. Google Scholar
Digital Library
- MS Bartlett, G Littlewort, C Lainscsek, I Fasel, and J Movellan. 2004. Machine learning methods for fully automatic recognition of facial expressions and facial actions. In Proc. of International Conference on Systems, Man and Cybernetics (SMC '04), 592--597.Google ScholarCross Ref
- MS Bartlett, G Littlewort, M Frank, C Lainscsek, I Fasel, and J Movellan. 2005. Recognizing facial expression: Machine learning and application to spontaneous behavior. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR '05), 568--573. Google ScholarDigital Library
- KP Bennett, M Momma, and MJ Embrechts. 2002. MARK: A boosting algorithm for heterogeneous kernel models. In Proc. of the 8th International Conference on Knowledge Discovery and Data mining (KDD '02), 24--31. Google ScholarDigital Library
- CJC Burges. 1998. A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Discov. 2, 2 (June 1998), 121--167. Google ScholarDigital Library
- Y-W Chang, C-J Hsieh, K-W Chang, M Ringgaard, and C-J Lin. 2010. Training and testing low-degree polynomial data mappings via linear SVM. J. Mach. Learn. Res. 11 (August 2010), 1471--1490. Google ScholarDigital Library
- Z Cheng, Y Zhang, C Zhou, W Zhang and S Gao. 2009. Classification of 5--HT1A receptor ligands on the basis of their binding affinities by using PSO-Adaboost-SVM. Int. J. Mol. Sci. 10, 8 (July 2009), 3316--3337.Google ScholarCross Ref
- Z Cheng and Y Zhang. 2010. Classification models of estrogen receptor-β ligands based on PSO-Adaboost-SVM. Journal of Convergence Information Technology 5, 2 (April 2010), 67--83.Google Scholar
- C Cortes and V Vapnik. 1995. Support-vector networks. Mach. Learn. 20, 3 (September 1995), 273--297. Google ScholarCross Ref
- K Crammer, J Keshet, and Y Singer. 2002. Kernel design using boosting. In Proc. of Conference on Advances in Neural Information Processing Systems 15 (NIPS '02), 553--560. Google ScholarDigital Library
- Y-S Dong and K-S Han. 2005. Boosting SVM classifiers by ensemble. In Special Interest Tracks and Posters of the 14th International Conference on World Wide Web (WWW '05), 1072--1073. Google ScholarDigital Library
- H Drucker, D Wu, and V Vapnik. 1999. Support vector machines for spam categorization. Trans. Neur. Netw. 10, 5 (September 1999), 1048--1054. Google ScholarDigital Library
- KO Elish and MO Elish. 2008. Predicting defect-prone software modules using support vector machines. J. Syst. Softw. 81, 5 (May 2008), 649--660. Google ScholarDigital Library
- A Este, F Gringoli, and L Salgarelli. 2009. Support vector machines for TCP traffic classification. Comput. Netw. 53, 14 (September 2009), 2476--2490. Google ScholarDigital Library
- A Esuli, T Fagni, and F Sebastiani. 2006. MP-Boost: a multiple-pivot boosting algorithm and its application to text categorization. In Proc. of the 13th International Conference on String Processing and Information Retrieval (SPIRE '06), 1--12. Google ScholarDigital Library
- Y Freund and RE Schapire. 1995. A decision-theoretic generalization of on-line learning and an application to boosting. In Proc. of the 2nd European Conference on Computational Learning Theory (EuroCOLT '95), 23--37. Google ScholarDigital Library
- N Gkalelis and V Mezaris. 2014. Video event detection using generalized subclass discriminant analysis and linear support vector machines. In Proc. of International Conference on Multimedia Retrieval (ICMR '14), 25--32. Google ScholarDigital Library
- M Hall, E Frank, G Holmes, B Pfahringer, P Reutemann, and IH Witten. 2009. The WEKA data mining software: an update. SIGKDD Explor. Newsl. 11, 1 (November 2009), 10--18. Google ScholarDigital Library
- T Hastie and R Tibshirani. 1998. Classification by pairwise coupling. In Proc. of Conference on Advances in Neural Information Processing Systems 10 (NIPS '97), 507--513. Google ScholarDigital Library
- T Hertz, AB Hillel, and D Weinshall. 2006. Learning a kernel function for classification with small training samples. In Proc. of the 23rd International Conference on Machine learning (ICML '06), 401--408. Google ScholarDigital Library
- T Joachims. 2001. A statistical learning model of text classification for support vector machines. In Proc. of the 24th International Conference on Research and Development in Information Retrieval (SIGIR '01), 128--136. Google ScholarDigital Library
- SS Keerthi, SK Shevade, C Bhattacharyya, and KRK Murthy. 2001. Improvements to Platt's SMO algorithm for SVM classifier design. Neural Comput. 13, 3 (March 2001), 637--649. Google ScholarDigital Library
- L Khan, M Awad, and B Thuraisingham. 2007. A new intrusion detection system using support vector machines and hierarchical clustering. VLDB J. 16, 4 (October 2007), 507--521. Google ScholarDigital Library
- X Li, L Wang, and E Sung. 2005. A study of AdaBoost with SVM based weak learners. In Proc. of International Joint Conference on Neural Networks (IJCNN '05), 196--201.Google Scholar
- X Li, L Wang, and E Sung. 2008. AdaBoost with SVM-based component classifiers. Eng. Appl. Artif. Intell. 21, 5 (August 2008), 785--795. Google ScholarDigital Library
- M Lichman. 2013. UCI Machine Learning Repository {http://archive.ics.uci.edu/ml}. Irvine, CA: University of California, School of Information and Computer Science.Google Scholar
- GC Littlewort, MS Bartlett, J Chenu, I Fasel, T Kanda, H Ishiguro, and J Movellan. 2003. Towards social robots: Automatic evaluation of human-robot interaction by face detection and expression classification. In Proc. of Conference on Advances in Neural Information Processing Systems 16 (NIPS '03), 1563--1570. Google ScholarDigital Library
- Y Liu, A An, and X Huang. 2006. Boosting prediction accuracy on imbalanced datasets with SVM ensembles. In Proc. of the 10th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining (PAKDD '06), 107--118. Google ScholarDigital Library
- S Maldonado, Á Flores, T Verbraken, B Baesens, and R Weber. 2015. Profit-based feature selection using support vector machines - General framework and an application for customer retention. Appl. Soft Comput. 35, C (October 2015), 740--748. Google ScholarDigital Library
- P Michel and R El Kaliouby. 2003. Real time facial expression recognition in video using support vector machines. In Proc. of the 5th International Conference on Multimodal Interfaces (ICMI '03), 258--264. Google ScholarDigital Library
- M Namdari and H Jazayeri-Rad. 2014. Incipient fault diagnosis using support vector machines based on monitoring continuous decision functions. Eng. Appl. Artif. Intell. 28 (February 2014), 22--35. Google ScholarDigital Library
- K Nishida and T Kurita. 2005. Boosting soft-margin SVM with feature selection for pedestrian detection. In Proc. of the 6th international conference on Multiple Classifier Systems (MCS '05), 22--31. Google ScholarDigital Library
- N Nissim, R Moskovitch, L Rokach, and Y Elovici. 2012. Detecting unknown computer worm activity via support vector machines and active learning. Pattern Anal. Appl. 15, 4 (November 2012), 459--475. Google ScholarDigital Library
- E Owusu, Y Zhan, and QR Mao. 2014. An SVM-AdaBoost facial expression recognition system. Applied Intelligence 40, 3 (April 2014), 536--545. Google ScholarDigital Library
- D Pavlov, J Mao, and B Dom. 2000. Scaling-up support vector machines using boosting algorithm. In Proc. of the 15th International Conference on Pattern Recognition (ICPR '00), 219--222.Google ScholarCross Ref
- JC Platt. 1999. Fast training of support vector machines using sequential minimal optimization. In Advances in Kernel Methods. MIT Press, 185--208. Google ScholarDigital Library
- WH Press, SA Teukolsky, WT Vetterling, and BP Flannery. 2007. Numerical Recipes: The Art of Scientific Computing (3rd Ed.). Cambridge University Press.Google ScholarDigital Library
- A Savio, M García-Sebastián, M Graña, and J Villanúa. 2009. Results of an Adaboost Approach on Alzheimer's Disease Detection on MRI. In Proc. of the 3rd International Work-Conference on the Interplay Between Natural and Artificial Computation: Part II: Bioinspired Applications in Artificial and Natural Computation (IWINAC '09), 114--123. Google ScholarDigital Library
- B Scholkopf and AJ Smola. 2001. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press. Google ScholarDigital Library
- H Schwenk and Y Bengio. 1997. AdaBoosting Neural Networks: Application to on-line Character Recognition. In Proc. of the 7th International Conference on Artificial Neural Networks (ICANN '97), 967--972. Google ScholarDigital Library
- F Sebastiani, A Sperduti, and N Valdambrini. 2000. An improved boosting algorithm and its application to text categorization. In Proc. of the 9th International Conference on Information and Knowledge Management (CIKM '00), 78--85. Google ScholarDigital Library
- A Sharma and S Dey. 2013. A boosted SVM based sentiment analysis approach for online opinionated text. In Proc. of Research in Adaptive and Convergent Systems (RACS '13), 28--34. Google ScholarDigital Library
- J Sun, M-Y Jia, and Hui Li. 2011. AdaBoost ensemble for financial distress prediction: An empirical comparison with data from Chinese listed companies. Expert Syst. Appl. 38, 8 (August 2011), 9305--9312. Google ScholarDigital Library
- X Tang, Z Ou, T Su, H Sun, and P Zhao. 2005. Robust precise eye location by adaboost and SVM techniques. In Proc. of the 2nd International Conference on Advances in Neural Networks (ISNN '05), 93--98. Google ScholarDigital Library
- B Tang, H Cao, Y Wu, M Jiang, and H Xu. 2012. Clinical entity recognition using structural support vector machines with rich features. In Proc. of the 6th International Workshop on Data and Text Mining in Biomedical Informatics (DTMBIO '12), 13--20. Google ScholarDigital Library
- B Üstün, WJ Melssen, and LMC Buydens. 2006. Facilitating the application of support vector regression by using a universal Pearson VII function based kernel. Chemometr. Intell. Lab. 81, 1 (March 2006), 29--40, 0169--7439Google ScholarCross Ref
- BX Wang and N Japkowicz. 2010. Boosting support vector machines for imbalanced data sets. Knowl. Inf. Syst. 25, 1 (October 2010), 1--20. Google ScholarDigital Library
- X Wu, V Kumar, JR Quinlan, J Ghosh, Q Yang, H Motoda, GJ McLachlan, A Ng, B Liu, PS Yu, Z-H Zhou, M Steinbach, DJ Hand, and D Steinberg. 2007. Top 10 algorithms in data mining. Knowl. Inf. Syst. 14, 1 (December 2007), 1--37. Google ScholarDigital Library
- X Xu and TS Huang. 2007. SODA-boosting and its application to gender recognition. In Proc. of the 3rd International Conference on Analysis and Modeling of Faces and Gestures (AMFG'07), 193--204. Google ScholarDigital Library
- D Zhang and WS Lee. 2004. Web taxonomy integration using support vector machines. In Proc. of the 13th International Conference on World Wide Web (WWW '04), 472--481. Google ScholarDigital Library
- J Zhu, H Zou, S Rosset, and T Hastie. 2009. Multi-class adaboost. Statistics and Its Interface 2 (2009), 349--360.Google ScholarCross Ref
Index Terms
- Integrating adaptive boosting and support vector machines with varying kernels
Recommendations
Pairwise multi-classification support vector machines: quadratic programming (QP-PAMSVM) formulations
NN'05: Proceedings of the 6th WSEAS international conference on Neural networksThe binary support vector machines (SVMs) have been extensively investigated. However their extension to a multi-classification model is still an on-going research. In this paper we present an extension of the binary support vector machines (SVMs) for ...
Fast instance selection for speeding up support vector machines
Support vector machine (SVM) has shown prominent performance for binary classification. How to effectively apply it to massive datasets with large number of classes and instances is still a serious challenge. Instance selection methods have been ...
Adaboost with SVM-based classifier for the classification of brain motor imagery tasks
UAHCI'11: Proceedings of the 6th international conference on Universal access in human-computer interaction: users diversity - Volume Part IIThe Adaboost with SVM-based component classifier is generally considered to break the Boosting principle for the difficulty in training of SVM and have imbalance between the diversity and accuracy over basic SVM classifiers. The Adaboost classifier in ...
Comments