Abstract
Boosting has established itself as a successful technique for decreasing the generalization error of classification learners by basing predictions on ensembles of hypotheses. While previous research has shown that this technique can be made to work efficiently even in the context of multirelational learning by using simple learners and active feature selection, such approaches have relied on simple and static methods of determining feature selection ordering a priori and adding features only in a forward manner. In this paper, we investigate whether the distributional information present in boosting can usefully be exploited in the course of learning to reweight features and in fact even to dynamically adapt the feature set by adding the currently most relevant features and removing those that are no longer needed. Preliminary results show that these more informed feature set evolution strategies surprisingly have mixed effects on the number of features ultimately used in the ensemble, and on the resulting classification accuracy.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Berka, P.: Guide to the financial Data Set. In: Siebes, A., Berka, P. (eds.) PKDD 2000 Discovery Challenge (2000)
Cheng, J., Hatzis, C., Hayashi, H., Krogel, M.-A., Morishita, S., Page, D., Sese, J.: KDD Cup 2001 Report. SIGKDD Explorations 3(2), 47–64 (2002)
Cohen, W., Singer, Y.: A Simple, Fast, and Effective Rule Learner. In: Proc. of 16th National Conference on Artificial Intelligence (1999)
Das, S.: Filters, Wrappers and a Boosting-based Hybrid for Feature Selection. In: Proc. of 18th International Conference on Machine Learning (2001)
Freund, Y., Schapire, R.E.: Experiments with a New Boosting Algorithm. In: Proc. of 13th International Conference on Machine Learning (1996)
Grove, A.J., Schuurmans, D.: Boosting in the limit: Maximizing the margin of learned ensembles. In: Proc. of 15th National Conf. on AI (1998)
Hoche, S., Wrobel, S.: Relational Learning Using Constrained Confidence-Rated Boosting. In: Rouveirol, C., Sebag, M. (eds.) ILP 2001. LNCS (LNAI), vol. 2157, p. 51. Springer, Heidelberg (2001)
Hoche, S., Wrobel, S.: Scaling Boosting by Margin-Based Inclusion of Featuresand Relations. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS (LNAI), vol. 2430, p. 148. Springer, Heidelberg (2002)
King, R.D., Muggleton, S., Lewis, R.A., Sternberg, M.J.E.: Drug design by machine learning: The use of inductive logic programming to model the structure activity relationships of trimethoprim analogues binding to dihydrofolate reductase. Proc. of the National Academy of Sciences of the USA 89(23), 11322–11326 (1992)
King, R.D., Srinivasan, A., Sternberg, M.: Relating chemical activity to structure: An examination of ILP successes. New Generation Computing, Special issue on Inductive Logic Programming 13(3-4), 411–434 (1995)
Kramer, S., De Raedt, L.: Feature construction with version spaces for biochemical applications. In: Proc. of the 18th ICML (2001)
Kramer, S.: Demand-driven Construction of Structural Features in ILP. In: Rouveirol, C., Sebag, M. (eds.) ILP 2001. LNCS (LNAI), vol. 2157, p. 132. Springer, Heidelberg (2001)
McGill, W.J.: Multivariate information transmission. IRE Trans. Inf. Theory (1995)
Opitz, D., Maclin, R.: Popular Ensemble Method: An Empirical Study. Journal of Artificial Intelligence Research 11, 169–198 (1999)
Quinlan, J.R.: Bagging, boosting, and C4.5. In: Proc. of 14th Nat. Conf. on AI (1996)
Schapire, R.E.: Theoretical views of boosting and applications. In: Proceedings of the 10th International Conference on Algorithmic Learning Theory (1999)
Sebban, M., Nock, R.: Contribution of Boosting in Wrapper Models. In: Żytkow, J.M., Rauch, J. (eds.) PKDD 1999. LNCS (LNAI), vol. 1704, pp. 214–222. Springer, Heidelberg (1999)
Schapire, R.E., Freund, Y., Bartlett, P., Lee, W.S.: Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods. The Annals of Statistics 26(5), 1651–1686 (1998)
Shannon, C.E.: A mathematical theory of communication. Bell. Syst. Techn. J. 27, 379–423 (1948)
Srinivasan, A., Muggleton, S., Sternberg, M.J.E., King, R.D.: Theories for mutagenicity: A study in first-order and feature-based induction. Artificial Intelligence (1996)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Hoche, S., Wrobel, S. (2003). A Comparative Evaluation of Feature Set Evolution Strategies for Multirelational Boosting. In: Horváth, T., Yamamoto, A. (eds) Inductive Logic Programming. ILP 2003. Lecture Notes in Computer Science(), vol 2835. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-39917-9_13
Download citation
DOI: https://doi.org/10.1007/978-3-540-39917-9_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-20144-1
Online ISBN: 978-3-540-39917-9
eBook Packages: Springer Book Archive