skip to main content
10.1145/3459637.3483279acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
tutorial

AutoML: From Methodology to Application

Published: 30 October 2021 Publication History

Abstract

Machine Learning methods have been adopted for a wide range of real-world applications, ranging from social networks, online image/video-sharing platforms, and e-commerce to education, healthcare, etc. However, in practice, a large amount of effort is required to tune several components of machine learning methods, including data representation, hyperparameter, and model architecture, in order to achieve a good performance. To alleviate the required tunning efforts, Automated Machine Learning (AutoML), which can automate the process of applying machine learning methods, has been studied in both academy and industry recently. In this tutorial, we will introduce the main research topics of AutoML, including Hyperparameter Optimization, Neural Architecture Search, and Meta-Learning. Two emerging topics of AutoML, Automatic Feature Generation and Machine Learning Guided Database, will also be discussed since they are important components for real-world applications. For each topic, we will motivate it with application examples from industry, illustrate the state-of-the-art methodologies, and discuss some future research directions based on our experience from industry and the trends in academy.

References

[1]
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understanding and simplifying one-shot architecture search. In Proceedings of the 35th International Conference on Machine Learning, pages 550--559, 2018.
[2]
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 2020.
[3]
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations, 2019.
[4]
Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou. Adabert: Task-adaptive BERT compression with differentiable neural architecture search. In Proc. of the International Jont Conference on Artifical Intelligence, pages 2463--2469, 2020.
[5]
Jialin Ding, Umar Farooq Minhas, Hantian Zhang, Yinan Li, Chi Wang, Badrish Chandramouli, Johannes Gehrke, Donald Kossmann, and David B. Lomet. Alex: An updatable adaptive learned index. page 969--984, 2020.
[6]
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. NATS-Bench: Benchmarking nas algorithms for architecture topology and size. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.
[7]
Anshuman Dutt, Chi Wang, Azade Nazi, Srikanth Kandula, Vivek Narasayya, and Surajit Chaudhuri. Selectivity estimation for range predicates using lightweight models. Proceedings of the VLDB Endowment, 12(9):1044--1057, 2019.
[8]
Paolo Ferragina and Giorgio Vinciguerra. The PGM-index: a fully-dynamic compressed learned index with provable worst-case bounds. volume 13, pages 1162--1175, 2020.
[9]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126--1135, 2017.
[10]
Alex Galakatos, Michael Markovitch, Carsten Binnig, Rodrigo Fonseca, and Tim Kraska. FITing-Tree: A data-aware index structure. In Proc. of the ACM SIGMOD International Conference on Management of Data, pages 1189--1206, 2019.
[11]
Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu, and Dahua Lin. When nas meets robustness: In search of robust architectures against adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[12]
Benjamin Hilprecht, Andreas Schmidt, Moritz Kulessa, Alejandro Molina, Kristian Kersting, and Carsten Binnig. Deepdb: Learn from data, not from queries! arXiv preprint arXiv:1909.00607, 2019.
[13]
Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Artificial Intelligence and Statistics, pages 240--248, 2016.
[14]
Gilad Katz, Eui Chul Richard Shin, and Dawn Song. Explorekit: Automatic feature generation and selection. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pages 979--984. IEEE, 2016.
[15]
Andreas Kipf, Thomas Kipf, Bernhard Radke, Viktor Leis, Peter Boncz, and Alfons Kemper. Learned cardinalities: Estimating correlated joins with deep learning. arXiv preprint arXiv:1809.00677, 2018.
[16]
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, 2015.
[17]
Tim Kraska, Alex Beutel, Ed H Chi, Jeffrey Dean, and Neoklis Polyzotis. The case for learned index structures. In Proc. of the ACM SIGMOD International Conference on Management of Data, pages 489--504, 2018.
[18]
Ang Li, Ola Spyra, Sagi Perel, Valentin Dalibard, Max Jaderberg, Chenjie Gu, David Budden, Tim Harley, and Pramod Gupta. A generalized framework for population based training. In Proc. of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1791--1799, 2019.
[19]
Liam Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18--185:1--52, 2018.
[20]
Yaliang Li, Daoyuan Chen, Bolin Ding, Kai Zeng, and Jingren Zhou. A pluggable learned index method via sampling and gap insertion. arXiv preprint arXiv:2101.00808, 2021.
[21]
Zekun Li, Zeyu Cui, Shu Wu, Xiaoyu Zhang, and Liang Wang. Fi-GNN: Modeling feature interactions via graph neural networks for ctr prediction. In Proc. of the International Conference on Information and Knowledge Management, pages 539--548, 2019.
[22]
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations (ICLR), 2019.
[23]
Yuanfei Luo, Mengshuo Wang, Hao Zhou, Quanming Yao, Wei-Wei Tu, Yuqiang Chen, Wenyuan Dai, and Qiang Yang. Autocross: Automatic feature crossing for tabular data in real-world applications. In Proc. of the SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1936--1945, 2019.
[24]
Matthew MacKay, Paul Vicol, Jonathan Lorraine, David Duvenaud, and Roger Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. In International Conference on Learning Representations (ICLR), 2019.
[25]
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, M. Alizadeh, T. Kraska, Olga Papaemmanouil, and Nesime Tatbul. Neo: A learned query optimizer. Proc. VLDB Endow., 12:1705--1718, 2019.
[26]
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999, 2018.
[27]
Asaf Noy, Niv Nayman, Tal Ridnik, Nadav Zamir, Sivan Doveh, Itamar Friedman, Raja Giryes, and Lihi Zelnik. Asap: Architecture search, anneal and prune. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, 2020.
[28]
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Proc. of the International Conference on Machine Learning, pages 4092--4101, 2018.
[29]
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollar. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[30]
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning, pages 1842--1850, 2016.
[31]
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, pages 2951--2959, 2012.
[32]
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. AutoInt: Automatic feature interaction learning via self-attentive neural networks. In Proc. of the International Conference on Information and Knowledge Management, pages 1161--1170, 2019.
[33]
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105--6114, 2019.
[34]
Zhiqiang Tao, Yaliang Li, Bolin Ding, Ce Zhang, Jingren Zhou, and Yun Fu. Learning to mutate with hypergradient guided population. Advances in Neural Information Processing Systems, 33, 2020.
[35]
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, pages 3630--3638, 2016.
[36]
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection in differentiable NAS. In International Conference on Learning Representations, 2021.
[37]
Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. Exploring randomly wired neural networks for image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[38]
Yuexiang Xie, Zhen Wang, Yaliang Li, Bolin Ding, Nezihe Merve Gürel, Ce Zhang, Minlie Huang, Wei Lin, and Jingren Zhou. Interactive feature generation via learning adjacency tensor of feature graph. arXiv preprint arXiv:2007.14573, 2020.
[39]
Zongheng Yang, Amog Kamsetty, Sifei Luan, Eric Liang, Yan Duan, Xi Chen, and Ion Stoica. Neurocard: one cardinality estimator for all tables. arXiv preprint arXiv:2006.08109, 2020.
[40]
Zongheng Yang, Eric Liang, Amog Kamsetty, Chenggang Wu, Yan Duan, Xi Chen, Pieter Abbeel, Joseph M Hellerstein, Sanjay Krishnan, and Ion Stoica. Selectivity estimation with deep likelihood models. arXiv preprint arXiv:1905.04278, 2019.
[41]
Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning. In International Conference on Machine Learning, pages 7045--7054, 2019.
[42]
Huaxiu Yao, Xian Wu, Zhiqiang Tao, Yaliang Li, Bolin Ding, Ruirui Li, and Zhenhui Li. Automated relational meta-learning. In International Conference on Learning Representations, 2020.
[43]
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS-bench-101: Towards reproducible neural architecture search. In Proceedings of the 36th International Conference on Machine Learning, pages 7105--7114, 2019.
[44]
Jiaxuan You, Jure Leskovec, Kaiming He, and Saining Xie. Graph structure of neural networks. In Proc. of the International Conference on Machine Learning, pages 10881--10891, 2020.
[45]
Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. In Advances in Neural Information Processing Systems, 2020.
[46]
Yuge Zhang, Zejun Lin, Junyang Jiang, Quanlu Zhang, Yujing Wang, Hui Xue, Chen Zhang, and Yaming Yang. Deeper insights into weight sharing in neural architecture search. arXiv preprint arXiv:2001.01431, 2020.
[47]
Kaixiong Zhou, Qingquan Song, Xiao Huang, and Xia Hu. Auto-gnn: Neural architecture search of graph neural networks. arXiv preprint arXiv:1909.03184, 2019.
[48]
Rong Zhu, Ziniu Wu, Yuxing Han, Kai Zeng, Andreas Pfadler, Zhengping Qian, Jingren Zhou, and Bin Cui. Flat: Fast, lightweight and accurate method for cardinality estimation. arXiv preprint arXiv:2011.09022, 2020.
[49]
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. International Conference on Learning Representations (ICLR), 2017.

Cited By

View all
  • (2024)How Automated Machine Learning Can Improve BusinessApplied Sciences10.3390/app1419874914:19(8749)Online publication date: 27-Sep-2024
  • (2024)Unsupervised Generative Feature Transformation via Graph Contrastive Pre-training and Multi-objective Fine-tuningProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3672015(3966-3976)Online publication date: 25-Aug-2024
  • (2024)Zero-touch networksComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2024.110294243:COnline publication date: 1-Apr-2024
  • Show More Cited By

Index Terms

  1. AutoML: From Methodology to Application

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management
    October 2021
    4966 pages
    ISBN:9781450384469
    DOI:10.1145/3459637
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 30 October 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Qualifiers

    • Tutorial

    Conference

    CIKM '21
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

    Upcoming Conference

    CIKM '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)77
    • Downloads (Last 6 weeks)8
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)How Automated Machine Learning Can Improve BusinessApplied Sciences10.3390/app1419874914:19(8749)Online publication date: 27-Sep-2024
    • (2024)Unsupervised Generative Feature Transformation via Graph Contrastive Pre-training and Multi-objective Fine-tuningProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3672015(3966-3976)Online publication date: 25-Aug-2024
    • (2024)Zero-touch networksComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2024.110294243:COnline publication date: 1-Apr-2024
    • (2024)How far are we with automated machine learning? characterization and challenges of AutoML toolkitsEmpirical Software Engineering10.1007/s10664-024-10450-y29:4Online publication date: 13-Jun-2024
    • (2024)Context Aware Auto-AI FrameworkAdvances in Information and Communication10.1007/978-3-031-53963-3_44(639-645)Online publication date: 17-Mar-2024
    • (2023)Reinforcement-enhanced autoregressive feature transformationProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668009(43563-43578)Online publication date: 10-Dec-2023
    • (2023)Performance of Automated Machine Learning Based Neural Network Estimators for the Classification of PCOSIntelligent Human Centered Computing10.1007/978-981-99-3478-2_7(65-73)Online publication date: 15-Jun-2023
    • (2022)Cost-Sensitive Metaheuristic Optimization-Based Neural Network with Ensemble Learning for Financial Distress PredictionApplied Sciences10.3390/app1214691812:14(6918)Online publication date: 8-Jul-2022
    • (2022)A General Recipe for Automated Machine Learning in PracticeAdvances in Artificial Intelligence – IBERAMIA 202210.1007/978-3-031-22419-5_21(243-254)Online publication date: 23-Nov-2022
    • (2021)Hadoop-MTA: a system for Multi Data-center Trillion Concepts Auto-ML atop Hadoop2021 IEEE International Conference on Big Data (Big Data)10.1109/BigData52589.2021.9671617(5953-5955)Online publication date: 15-Dec-2021

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media