Abstract
We introduce a broad learning model formed by cascades of predictors, Deep Cascades, that is structured as general decision trees in which leaf predictors or node questions may be members of rich function families. We present new data-dependent theoretical guarantees for learning with Deep Cascades with complex leaf predictors and node questions in terms of the Rademacher complexities of the sub-families composing these sets of predictors and the fraction of sample points reaching each leaf that are correctly classified. These guarantees can guide the design of a variety of different algorithms for deep cascade models and we give a detailed description of two such algorithms. Our second algorithm uses as node and leaf classifiers SVM predictors and we report the results of experiments comparing its performance with that of SVM combined with polynomial kernels.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Arreola, K., Fehr, J., Burkhardt, H.: Fast support vector machine classification using linear SVMs. In: ICPR (2006)
Arreola, K., Fehr, J., Burkhardt, H.: Fast support vector machine classification of very large datasets. In: GfKl Conference (2007)
Bartlett, P., Mendelson, S.: Rademacher and Gaussian complexities: Risk bounds and structural results. JMLR (2002)
Bengio, S., Weston, J., Weston, D.: Label embedding trees for large multi-class tasks. In: NIPS, Vancouver, Canada (2010)
Bennet, K., Blue, J.: A support vector machine approach to decision trees. In: IJCNN, Anchorage, Alaska (1998)
Breiman, L., Friedman, J., Olshen, R., Stone, C.: Classification and Regression Trees. Wadsworth and Brooks, Monterey (1984)
Chang, F., Guo, C., Lin, X., Lu, C.: Tree decomposition for large-scale SVM problems. JMLR (2010)
Chen, M., Xu, Z., Kedem, D., Chapelle, O.: Classifier cascade for minimizing feature evaluation cost. In: AISTATS, La Palma, Canary Islands (2012)
Cortes, C., Mohri, M., Syed, U.: Deep boosting. In: ICML (2014)
Deng, J., Satheesh, S., Berg, A., Fei-Fei, L.: Fast and balanced: efficient label tree learning for large scale object recognition. In: NIPS (2011)
DeSalvo, G., Mohri, M., Syed, U.: Learning with Deep Cascades. arXiv (2015)
Dong, G., Chen, J.: Study on support vector machine based decision tree and application. In: ICNC-FSKD, Jinan, China (2008)
Golea, M., Bartlett, P., Lee, W., Mason, L.: Generalization in decision trees and DNF: does size matter? In: NIPS (1997)
Koltchinskii, V., Panchenko, D.: Empirical margin distributions and bounding the generalization error of combined classifiers. Annals of Statistics 30 (2002)
Kumar, A., Gopal, M.: A hybrid SVM based decision tree. JPR (2010)
Lefakis, L., Fleuret, F.: Joint cascade optimization using a product of boosted classifiers. In: NIPS (2010)
Littman, M., Li, L., Walsh, T.: Knows what it knows: a framework for self-aware learning. In: ICML (2008)
Madjarov, G., Gjorgjevikj, D.: Hybrid decision tree architecture utilizing local SVMs for multi-label classification. In: Corchado, E., Snášel, V., Abraham, A., Woźniak, M., Graña, M., Cho, S.-B. (eds.) HAIS 2012, Part II. LNCS, vol. 7209, pp. 1–12. Springer, Heidelberg (2012)
Mansour, Y., McAllester, D.: Generalization bounds for decision trees. In: COLT (2000)
Mohri, M., Rostamizadeh, R., Talwalkar, A.: Foundations of Machine Learning. The MIT Press (2012)
Nobel, A.: Analysis of a complexity based pruning scheme for classification trees. IEEE Trans. Inf. Theory (2002)
Pujara, J., Daume, H., Getoor, L.: Using classifier cascades for scalable e-mail classification. In: CEAS (2011)
Quinlan, J.: Induction of decision trees. Machine Learning 1(1), 81–106 (1986)
Rodriguez-Lujan, I., Cruz, C., Huerta, R.: Hierarchical linear SVM. JPR (2012)
Saberian, M., Vasconcelos, N.: Boosting classifier cascades. In: NIPS, Canada (2010)
Scott, C., Nowak, R.: On adaptive properties of decision trees. In: NIPS, Canada (2005)
Takahashi, F., Abe, S.: Decision tree based multiclass SVMs. In: ICONIP (2002)
Viola, P., Jones, M.: Robust real-time face detection. IJCV (2004)
Wang, J., Saligrama, V.: Local supervised learning through space partitioning. In: NIPS (2012)
Xu, Z., Kusner, M., Weinberger, K., Chen, M.: Cost-sensitive tree of classifiers. In: ICML, Altanta, USA (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
DeSalvo, G., Mohri, M., Syed, U. (2015). Learning with Deep Cascades. In: Chaudhuri, K., GENTILE, C., Zilles, S. (eds) Algorithmic Learning Theory. ALT 2015. Lecture Notes in Computer Science(), vol 9355. Springer, Cham. https://doi.org/10.1007/978-3-319-24486-0_17
Download citation
DOI: https://doi.org/10.1007/978-3-319-24486-0_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-24485-3
Online ISBN: 978-3-319-24486-0
eBook Packages: Computer ScienceComputer Science (R0)