skip to main content
10.1145/3447548.3470827acmconferencesArticle/Chapter ViewAbstractPublication PageskddConference Proceedingsconference-collections
abstract

AutoML: A Perspective where Industry Meets Academy

Published: 14 August 2021 Publication History

Abstract

Machine learning methods have been adopted for various real-world applications, ranging from social networks, online image/video-sharing platforms, and e-commerce to education, healthcare, etc. However, several components of machine learning methods, including data representation, hyperparameter and model architecture, can largely affect their performance in practice. Moreover, the explosions of data scale and model size make the optimization of these components more and more time-consuming for machine learning developers. To tackle these challenges, Automated Machine Learning (AutoML) aims to automate the process of applying machine learning methods to solve real-world application tasks, reducing the time of tuning machine learning methods while maintaining good performance. In this tutorial, we will introduce the main research topics of AutoML, including Hyperparameter Optimization, Neural Architecture Search and Meta-Learning. Two emerging topics of AutoML, DNN-based Feature Generation and Machine Learning Guided Database, will also be discussed as they are important components for real-world applications. For each topic, we will motivate it with examples from industry, illustrate the state-of-the-art methods, and discuss their pros and cons from both perspectives of industry and academy. We will also discuss some future research directions based on our experience from industry and the trends in academy.

References

[1]
Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou. Adabert: Task-adaptive BERT compression with differentiable neural architecture search. In Proc. of the International Jont Conference on Artifical Intelligence, pages 2463--2469, 2020.
[2]
Paolo Ferragina and Giorgio Vinciguerra. The PGM-index: a fully-dynamic compressed learned index with provable worst-case bounds. Proc. VLDB Endow., 13(8):1162--1175, 2020.
[3]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126--1135, 2017.
[4]
Alex Galakatos, Michael Markovitch, Carsten Binnig, Rodrigo Fonseca, and Tim Kraska. FITing-Tree: A data-aware index structure. In Proc. of the ACM SIGMOD International Conference on Management of Data, pages 1189--1206, 2019.
[5]
Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Artificial Intelligence and Statistics, pages 240--248, 2016.
[6]
Tim Kraska, Alex Beutel, Ed H Chi, Jeffrey Dean, and Neoklis Polyzotis. The case for learned index structures. In Proc. of the ACM SIGMOD International Conference on Management of Data, pages 489--504, 2018.
[7]
Ang Li, Ola Spyra, Sagi Perel, Valentin Dalibard, Max Jaderberg, Chenjie Gu, David Budden, Tim Harley, and Pramod Gupta. A generalized framework for population based training. In Proc. of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1791--1799, 2019.
[8]
Liam Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18--185:1--52, 2018.
[9]
Yaliang Li, Daoyuan Chen, Bolin Ding, Kai Zeng, and Jingren Zhou. A pluggable learned index method via sampling and gap insertion. arXiv preprint arXiv:2101.00808, 2021.
[10]
Zekun Li, Zeyu Cui, Shu Wu, Xiaoyu Zhang, and Liang Wang. Fi-GNN: Modeling feature interactions via graph neural networks for ctr prediction. In Proc. of the International Conference on Information and Knowledge Management, pages 539--548, 2019.
[11]
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations (ICLR), 2019.
[12]
Yuanfei Luo, Mengshuo Wang, Hao Zhou, Quanming Yao, Wei-Wei Tu, Yuqiang Chen, Wenyuan Dai, and Qiang Yang. Autocross: Automatic feature crossing for tabular data in real-world applications. In Proc. of the SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1936--1945, 2019.
[13]
Matthew MacKay, Paul Vicol, Jonathan Lorraine, David Duvenaud, and Roger Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. In International Conference on Learning Representations (ICLR), 2019.
[14]
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, M. Alizadeh, T. Kraska, Olga Papaemmanouil, and Nesime Tatbul. Neo: A learned query optimizer. Proc. VLDB Endow., 12:1705--1718, 2019.
[15]
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations (ICLR), 2018.
[16]
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999, 2018.
[17]
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Proc. of the International Conference on Machine Learning, pages 4092--4101, 2018.
[18]
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollar. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[19]
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, pages 2951--2959, 2012.
[20]
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. AutoInt: Automatic feature interaction learning via self-attentive neural networks. In Proc. of the International Conference on Information and Knowledge Management, pages 1161--1170, 2019.
[21]
Zhiqiang Tao, Yaliang Li, Bolin Ding, Ce Zhang, Jingren Zhou, and Yun Fu. Learning to mutate with hypergradient guided population. Advances in Neural Information Processing Systems, 33, 2020.
[22]
Ziniu Wu, Peilun Yang, Pei Yu, Rong Zhu, Yuxing Han, Yaliang Li, Defu Lian, Kai Zeng, and Jingren Zhou. A unified transferable model for ml-enhanced dbms. arXiv preprint arXiv:2105.02418, 2021.
[23]
Yuexiang Xie, Zhen Wang, Yaliang Li, Bolin Ding, Nezihe Merve Gürel, Ce Zhang, Minlie Huang, Wei Lin, and Jingren Zhou. Fives: Feature interaction via edge search for large-scale tabular data. In Proc. of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2021.
[24]
Zongheng Yang, Eric Liang, Amog Kamsetty, Chenggang Wu, Yan Duan, Xi Chen, Pieter Abbeel, Joseph M Hellerstein, Sanjay Krishnan, and Ion Stoica. Selectivity estimation with deep likelihood models. arXiv preprint arXiv:1905.04278, 2019.
[25]
Huaxiu Yao, Xian Wu, Zhiqiang Tao, Yaliang Li, Bolin Ding, Ruirui Li, and Zhenhui Li. Automated relational meta-learning. In International Conference on Learning Representations (ICLR), 2020.
[26]
Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. In Advances in Neural Information Processing Systems, 2020.
[27]
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. International Conference on Learning Representations (ICLR), 2017.

Cited By

View all
  • (2024)Time Series Representation for Visualization in Apache IoTDBProceedings of the ACM on Management of Data10.1145/36392902:1(1-26)Online publication date: 26-Mar-2024
  • (2023)Towards Flexible and Adaptive Neural Process for Cold-Start RecommendationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.330483936:4(1815-1828)Online publication date: 23-Aug-2023
  • (2023)AutoMLP: A Framework for the Acceleration of Multi-Layer Perceptron Models on FPGAs for Real-Time Atrial Fibrillation Disease DetectionIEEE Transactions on Biomedical Circuits and Systems10.1109/TBCAS.2023.329908417:6(1371-1386)Online publication date: Dec-2023
  • Show More Cited By

Index Terms

  1. AutoML: A Perspective where Industry Meets Academy

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining
    August 2021
    4259 pages
    ISBN:9781450383325
    DOI:10.1145/3447548
    Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 14 August 2021

    Check for updates

    Author Tags

    1. automl
    2. hyperparameter optimization
    3. meta-learning
    4. neural architecture search

    Qualifiers

    • Abstract

    Conference

    KDD '21
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

    Upcoming Conference

    KDD '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)16
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Time Series Representation for Visualization in Apache IoTDBProceedings of the ACM on Management of Data10.1145/36392902:1(1-26)Online publication date: 26-Mar-2024
    • (2023)Towards Flexible and Adaptive Neural Process for Cold-Start RecommendationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.330483936:4(1815-1828)Online publication date: 23-Aug-2023
    • (2023)AutoMLP: A Framework for the Acceleration of Multi-Layer Perceptron Models on FPGAs for Real-Time Atrial Fibrillation Disease DetectionIEEE Transactions on Biomedical Circuits and Systems10.1109/TBCAS.2023.329908417:6(1371-1386)Online publication date: Dec-2023
    • (2023)Automated Machine Learning (AutoML): The Future of Computational IntelligenceInternational Conference on Cyber Security, Privacy and Networking (ICSPN 2022)10.1007/978-3-031-22018-0_28(309-317)Online publication date: 21-Feb-2023
    • (2022)Automated Machine Learning & Tuning with FLAMLProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3534678.3542636(4828-4829)Online publication date: 14-Aug-2022
    • (2022)Transfer Learning based Search Space Design for Hyperparameter TuningProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3534678.3539369(967-977)Online publication date: 14-Aug-2022
    • (2022)FederatedScope-GNN: Towards a Unified, Comprehensive and Efficient Package for Federated Graph LearningProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3534678.3539112(4110-4120)Online publication date: 14-Aug-2022
    • (2022)Challenges in Deploying Machine Learning: A Survey of Case StudiesACM Computing Surveys10.1145/353337855:6(1-29)Online publication date: 7-Dec-2022
    • (2022)A General Recipe for Automated Machine Learning in PracticeAdvances in Artificial Intelligence – IBERAMIA 202210.1007/978-3-031-22419-5_21(243-254)Online publication date: 23-Nov-2022

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media