research-article

Geometric Heuristics for Transfer Learning in Decision Trees

Authors:

Siddhesh Chaubal,

Mateusz Rzepecki,

Patrick K. Nicholson,

Guangyuan Piao,

Alessandra SalaAuthors Info & Claims

CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

Pages 151 - 160

https://doi.org/10.1145/3459637.3482259

Published: 30 October 2021 Publication History

Abstract

Motivated by a network fault detection problem, we study how recall can be boosted in a decision tree classifier, without sacrificing too much precision. This problem is relevant and novel in the context of transfer learning(TL), in which few target domain training samples are available. We define a geometric optimization problem for boosting the recall of a decision tree classifier, and show it is NP-hard. To solve it efficiently, we propose several near-linear time heuristics, and experimentally validate these heuristics in the context of TL. Our evaluation includes 7 public datasets, as well as 6 network fault datasets, and we compare our heuristics with several existing TL algorithms, as well as exact mixed integer linear programming(MILP) solutions to our optimization problem. We find that our heuristics boost recall in a manner similar to optimal MILP solutions, yet require several orders of magnitude less compute time. In many cases the F1 score of our approach is competitive, and often better, than other TL algorithms. Moreover, our approach can be used as a building block to apply transfer learning to more powerful ensemble methods, such as random forests.

Supplementary Material

MP4 File (CIKM21-rgfp0133.mp4)

Presentation video where we first describe our transfer learning setting followed by a formalization of the problem into a geometric optimization problem. We then describe our heuristic approaches to this problem, followed by experimental evaluations and results.

Download
33.89 MB

References

[1]

2020. https://github.com/TLDatasets/Datasets

[2]

Reem Al-Otaibi, Ricardo B. C. Prudêncio, Meelis Kull, and Peter Flach. 2015. Ver- satile Decision Trees for Learning Over Multiple Contexts. In Machine Learning and Knowledge Discovery in Databases. Springer International Publishing, Cham, 184--199.

Digital Library

[3]

Bogdan Armaselu and Ovidiu Daescu. 2017. Maximum Area Rectangle Separating Red and Blue Points. CoRR abs/1706.03268 (2017). arXiv:1706.03268

[4]

Arthur Asuncion and David Newman. 2007. UCI machine learning repository.

[5]

Jonathan Backer and J. Mark Keil. 2010. The Mono- and Bichromatic Empty Rectangle and Square Problems in All Dimensions. In LATIN 2010: Theoretical Informatics, Alejandro López-Ortiz (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 14--25.

Digital Library

[6]

Arturs Backurs, Nishanth Dikkala, and Christos Tzamos. 2016. Tight Hardness Results for Maximum Weight Rectangles. In 43rd International Colloquium on Automata, Languages, and Programming, ICALP 2016, July 11-15, 2016, Rome, Italy (LIPIcs, Vol. 55), Ioannis Chatzigiannakis, Michael Mitzenmacher, Yuval Rabani, and Davide Sangiorgi (Eds.). Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, 81:1--81:13.

[7]

Jérémy Barbay, Timothy M. Chan, Gonzalo Navarro, and Pablo Pérez-Lantero. 2014. Maximum-weight planar boxes in O(n2) time (and better). Inf. Process. Lett. 114, 8 (2014), 437--445.

[8]

Jon Louis Bentley. 1975. Multidimensional Binary Search Trees Used for Associa- tive Searching. Commun. ACM 18, 9 (1975), 509--517.

Digital Library

[9]

Dimitris Bertsimas and Jack Dunn. 2017. Optimal classification trees. Machine Learning 106, 7 (2017), 1039--1082.

Digital Library

[10]

Leo Breiman. 2001. Random Forests. Machine Learning 45, 1 (2001), 5--32.

Digital Library

[11]

Timothy M. Chan, Kasper Green Larsen, and Mihai Patrascu. 2011. Orthogonal range searching on the RAM, revisited. In Proceedings of the 27th ACM Symposium on Computational Geometry, Paris, France, June 13--15, 2011, Ferran Hurtado and Marc J. van Kreveld (Eds.). ACM, 1--10.

Digital Library

[12]

Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016. 785--794.

Digital Library

[13]

Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. 2009. Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems 47, 4 (2009), 547--553.

Digital Library

[14]

Hal Daumé III. 2007. Frustratingly Easy Domain Adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. Association for Computational Linguistics, Prague, Czech Republic, 256--263.

[15]

Jonathan Eckstein, Peter L. Hammer, Ying Liu, Mikhail Nediak, and Bruno Simeone. 2002. The Maximum Box Problem and its Application to Data Analysis. Comp. Opt. and Appl. 23, 3 (2002), 285--298.

Digital Library

[16]

Jeff Edmonds, Jarek Gryz, Dongming Liang, and Renée J Miller. 2003. Mining for empty spaces in large data sets. Theoretical Computer Science 296, 3 (2003), 435--452.

Digital Library

[17]

Peter W Frey and David J Slate. 1991. Letter recognition using Holland-style adaptive classifiers. Machine learning 6, 2 (1991), 161--182.

Digital Library

[18]

Jerzy W Grzymala-Busse, Linda K Goodwin, Witold J Grzymala-Busse, and Xinqun Zheng. 2004. An approach to imbalanced data sets based on changing rule strength. In Rough-neural computing. Springer, 543--553.

[19]

Jerzy W Grzymala-Busse, Jerzy Stefanowski, and Szymon Wilk. 2005. A comparison of two approaches to data mining from imbalanced data. Journal of Intelligent Manufacturing 16, 6 (2005), 565--573.

[20]

Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. 2009. The WEKA data mining software: an update. ACM SIGKDD explorations newsletter 11, 1 (2009), 10--18.

Digital Library

[21]

Xiyang Hu, Cynthia Rudin, and Margo Seltzer. 2019. Optimal sparse decision trees. In Advances in Neural Information Processing Systems. 7265--7273.

Digital Library

[22]

Sebastian Kauschke and Johannes Fürnkranz. 2018. Batchwise Patching of Clas- sifiers. In Thirty-Second AAAI Conference on Artificial Intelligence.

[23]

Jun Won Lee and Christophe G. Giraud-Carrier. 2007. Transfer Learning in Decision Trees. In Proceedings of the International Joint Conference on Neural Networks, IJCNN 2007, Celebrating 20 years of neural networks, Orlando, Florida, USA, August 12-17, 2007. 726--731.

[24]

Bing Liu, Liang-Ping Ku, and Wynne Hsu. 1997. Discovering interesting holes in data. In IJCAI (2). 930--935.

Digital Library

[25]

Wolfgang Maass. 1994. Efficient agnostic pac-learning with simple hypothesis. In Proceedings of the seventh annual conference on Computational learning theory. ACM, 67--75.

Digital Library

[26]

Sreerama K. Murthy. 1998. Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey. Data Min. Knowl. Discov. 2, 4 (1998), 345--389.

Digital Library

[27]

S. J. Pan and Q. Yang. 2010. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering 22, 10 (Oct 2010), 1345--1359. https://doi.org/10.1109/TKDE.2009.191

Digital Library

[28]

J. Ross Quinlan. 1987. Simplifying decision trees. International journal of man- machine studies 27, 3 (1987), 221--234.

Digital Library

[29]

Erik Rodner and Joachim Denzler. 2009. Learning with few examples by transfer- ring feature relevance. In Joint Pattern Recognition Symposium. Springer, 252--261.

[30]

Steven Salzberg. 1991. A nearest hyperrectangle learning method. Machine Learning 6, 3 (01 May 1991), 251--276. https://doi.org/10.1007/BF00114779

Digital Library

[31]

N. Segev, M. Harel, S. Mannor, K. Crammer, and R. El-Yaniv. 2017. Learn on Source, Refine on Target: A Model Transfer Learning Framework with Random Forests. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 9 (Sep. 2017), 1811--1824. https://doi.org/10.1109/TPAMI.2016.2618118

Digital Library

[32]

Jerzy Stefanowski and Daniel Vanderpooten. 2001. Induction of decision rules in classification and discovery-oriented perspectives. International Journal of Intelligent Systems 16, 1 (2001), 13--27.

[33]

Sicco Verwer and Y. Zhang. 2019. Learning optimal classification trees using a binary linear program formulation. Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19) AAAI-19 (2019), 1625,1632.

[34]

Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. 2016. A survey of transfer learning. Journal of Big data 3, 1 (2016), 9.

Cited By

Lu XWei L(2023)Teaching reform and innovation of vocational development and employment guidance courses in colleges and universities based on random forest modelApplied Mathematics and Nonlinear Sciences10.2478/amns.2023.2.002299:1Online publication date: 21-Aug-2023
https://doi.org/10.2478/amns.2023.2.00229

Index Terms

Geometric Heuristics for Transfer Learning in Decision Trees
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Multi-task learning
        Transfer learning
    2. Machine learning approaches
      1. Classification and regression trees

Recommendations

Selective sampling for trees and forests
Abstract
In this paper we describe selective sampling algorithms for decision trees and random forests and their contribution to the classification accuracy. In our selective sampling algorithms, the instance that yields the highest expected ...
Random Forests for land cover classification
Special issue: Pattern recognition in remote sensing (PRRS 2004)

Random Forests are considered for classification of multisource remote sensing and geographic data. Various ensemble classification methods have been proposed in recent years. These methods have been proven to improve classification accuracy ...
Semi-supervised classification trees

In many real-life problems, obtaining labelled data can be a very expensive and laborious task, while unlabeled data can be abundant. The availability of labeled data can seriously limit the performance of supervised learning methods. Here, we propose a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

October 2021

4966 pages

ISBN:9781450384469

DOI:10.1145/3459637

General Chairs:
Gianluca Demartini
The University of Queensland, Australia
,
Guido Zuccon
The University of Queensland, Australia
,
Program Chairs:
J. Shane Culpepper
RMIT University, Australia
,
Zi Huang
The University of Queensland, Australia
,
Hanghang Tong
University of Illinois at Urbana-Champaign, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM '21

Sponsor:

CIKM '21: The 30th ACM International Conference on Information and Knowledge Management

November 1 - 5, 2021

Queensland, Virtual Event, Australia

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
179
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lu XWei L(2023)Teaching reform and innovation of vocational development and employment guidance courses in colleges and universities based on random forest modelApplied Mathematics and Nonlinear Sciences10.2478/amns.2023.2.002299:1Online publication date: 21-Aug-2023
https://doi.org/10.2478/amns.2023.2.00229

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten