research-article

NAS-CTR: Efficient Neural Architecture Search for Click-Through Rate Prediction

Authors:

Yihua HuangAuthors Info & Claims

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 332 - 342

https://doi.org/10.1145/3477495.3532030

Published: 07 July 2022 Publication History

Abstract

Click-Through Rate (CTR) prediction has been widely used in many machine learning tasks such as online advertising and personalization recommendation. Unfortunately, given a domain-specific dataset, searching effective feature interaction operations and combinations from a huge candidate space requires significant expert experience and computational costs. Recently, Neural Architecture Search (NAS) has achieved great success in discovering high-quality network architectures automatically. However, due to the diversity of feature interaction operations and combinations, the existing NAS-based work that treats the architecture search as a black-box optimization problem over a discrete search space suffers from low efficiency. Therefore, it is essential to explore a more efficient architecture search method. To achieve this goal, we propose NAS-CTR, a differentiable neural architecture search approach for CTR prediction. First, we design a novel and expressive architecture search space and a continuous relaxation scheme to make the search space differentiable. Second, we formulate the architecture search for CTR prediction as a joint optimization problem with discrete constraints on architectures and leverage proximal iteration to solve the constrained optimization problem. Additionally, a straightforward yet effective method is proposed to eliminate the aggregation of skip connections. Extensive experimental results reveal that NAS-CTR can outperform the SOTA human-crafted architectures and other NAS-based methods in both test accuracy and search efficiency.

References

[1]

Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V Le. 2017. Neural optimizer search with reinforcement learning. arXiv preprint arXiv:1709.07417 (2017).

Digital Library

[2]

Han Cai, Ligeng Zhu, and Song Han. 2018. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332 (2018).

[3]

Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. 2019. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE International Conference on Computer Vision. 1294--1303.

[4]

Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. 7--10.

Digital Library

[5]

Xiangxiang Chu, Bo Zhang, and Xudong Li. 2020. Noisy differentiable architecture search. arXiv preprint arXiv:2005.03566 (2020).

[6]

Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. 2019. Fair darts: Eliminating unfair advantages in differentiable architecture search. arXiv preprint arXiv:1911.12126 (2019).

[7]

Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al. 2019. Neural architecture search: A survey. J. Mach. Learn. Res., Vol. 20, 55 (2019), 1--21.

[8]

Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction. arXiv preprint arXiv:1703.04247 (2017).

[9]

Xiangnan He and Tat-Seng Chua. 2017. Neural factorization machines for sparse predictive analytics. In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval. 355--364.

Digital Library

[10]

Gangwei Jiang, Hao Wang, Jin Chen, Haoyu Wang, Defu Lian, and Enhong Chen. 2021. xLightFM: Extremely Memory-Efficient Factorization Machine. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 337--346.

Digital Library

[11]

Manas R Joglekar, Cong Li, Mei Chen, Taibai Xu, Xiaoming Wang, Jay K Adams, Pranav Khaitan, Jiahui Liu, and Quoc V Le. 2020. Neural input search for large scale recommendation models. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2387--2397.

Digital Library

[12]

Defu Lian, Qi Liu, and Enhong Chen. 2020. Personalized Ranking with Importance Sampling. In Proceedings of The Web Conference 2020. 1093--1103.

Digital Library

[13]

Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. 2018. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1754--1763.

Digital Library

[14]

Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. 2019. Darts: Improved differentiable architecture search with early stopping. arXiv preprint arXiv:1909.06035 (2019).

[15]

Bin Liu, Niannan Xue, Huifeng Guo, Ruiming Tang, Stefanos Zafeiriou, Xiuqiang He, and Zhenguo Li. 2020 a. AutoGroup: Automatic feature grouping for modelling explicit high-order feature interactions in CTR prediction. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 199--208.

Digital Library

[16]

Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, Xiuqiang He, Zhenguo Li, and Yong Yu. 2020 c. AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction. arXiv preprint arXiv:2003.11235 (2020).

[17]

Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018).

[18]

Haochen Liu, Xiangyu Zhao, Chong Wang, Xiaobing Liu, and Jiliang Tang. 2020 b. Automated Embedding Size Search in Deep Recommender Systems. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2307--2316.

Digital Library

[19]

Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2018. Neural architecture optimization. In Advances in neural information processing systems. 7816--7827.

[20]

Yuanfei Luo, Mengshuo Wang, Hao Zhou, Quanming Yao, Wei-Wei Tu, Yuqiang Chen, Wenyuan Dai, and Qiang Yang. 2019. Autocross: Automatic feature crossing for tabular data in real-world applications. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1936--1945.

Digital Library

[21]

Jinnian Zhang, Yumeng Li, Jiancheng Li, Tanchao Zhu, and Lifeng Sun. 2021. A General Method For Automatic Discovery of Powerful Interactions In Click-Through Rate Prediction. In Proceedings of the 44th International Conference on Research and Development in Information Retrieval. 1298--1307.

[22]

Neal Parikh and Stephen Boyd. 2014. Proximal algorithms. Foundations and Trends in optimization, Vol. 1, 3 (2014), 127--239.

[23]

Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. 2018. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268 (2018).

[24]

Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. 2016. Product-based neural networks for user response prediction. In 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, 1149--1154.

[25]

Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alex Kurakin. 2017. Large-scale evolution of image classifiers. arXiv preprint arXiv:1703.01041 (2017).

Digital Library

[26]

Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. 2021. A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions. Comput. Surveys, Vol. 54, 4 (2021).

[27]

Steffen Rendle. 2010. Factorization machines. In 2010 IEEE International Conference on Data Mining. IEEE, 995--1000.

Digital Library

[28]

Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao. 2016. Deep crossing: Web-scale modeling without manually crafted combinatorial features. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 255--262.

Digital Library

[29]

Qingquan Song, Dehua Cheng, Hanning Zhou, Jiyan Yang, Yuandong Tian, and Xia Hu. 2020. Towards automated neural interaction discovery for click-through rate prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 945--955.

Digital Library

[30]

Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 1161--1170.

Digital Library

[31]

Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & cross network for ad click predictions. In Proceedings of the ADKDD'17. 1--7.

Digital Library

[32]

Ting-Hsiang Wang, Xia Hu, Haifeng Jin, Qingquan Song, Xiaotian Han, and Zirui Liu. 2020. AutoRec: An Automated Recommender System. In Fourteenth ACM Conference on Recommender Systems. 582--584.

[33]

Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, and Tat-Seng Chua. 2017. Attentional factorization machines: Learning the weight of feature interactions via attention networks. arXiv preprint arXiv:1708.04617 (2017).

[34]

Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. 2018. SNAS: stochastic neural architecture search. arXiv preprint arXiv:1812.09926 (2018).

[35]

Quanming Yao, Xiangning Chen, James T Kwok, Yong Li, and Cho-Jui Hsieh. 2020 a. Efficient neural interaction function search for collaborative filtering. In Proceedings of The Web Conference 2020. 1660--1670.

Digital Library

[36]

Quanming Yao, Ju Xu, Wei-Wei Tu, and Zhanxing Zhu. 2020 b. Efficient Neural Architecture Search via Proximal Iterations. In AAAI. 6664--6671.

[37]

Pengyu Zhao, Kecheng Xiao, Yuanxing Zhang, Kaigui Bian, and Wei Yan. 2020 b. AMER: Automatic Behavior Modeling and Interaction Exploration in Recommender System. arXiv preprint arXiv:2006.05933 (2020).

[38]

Xiangyu Zhao, Chong Wang, Ming Chen, Xudong Zheng, Xiaobing Liu, and Jiliang Tang. 2020 a. AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations. arXiv preprint arXiv:2002.11252 (2020).

[39]

Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Deep interest evolution network for click-through rate prediction. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 5941--5948.

Digital Library

[40]

Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1059--1068.

Digital Library

[41]

Jieming Zhu, Jinyang Liu, Shuai Yang, Qi Zhang, and Xiuqiang He. 2020. FuxiCTR: An Open Benchmark for Click-Through Rate Prediction. arXiv preprint arXiv:2009.05794 (2020).

[42]

Barret Zoph and Quoc V Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016).

Cited By

Jiang SZhu GWang YYuan CHuang YBaeza-Yates RBonchi F(2024)Automatic Multi-Task Learning Framework with Neural Architecture Search in RecommendationsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671715(1290-1300)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671715
Wu CLian DGe YZhou MChen ETao D(2024)Boosting Factorization Machines via Saliency-Guided MixupIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.335491046:6(4443-4459)Online publication date: Jun-2024
https://doi.org/10.1109/TPAMI.2024.3354910
Ding RXie RHao XYang XGe KZhang XZhou JLin L(2023)Interpretable User Retention Modeling in RecommendationProceedings of the 17th ACM Conference on Recommender Systems10.1145/3604915.3608818(702-708)Online publication date: 14-Sep-2023
https://dl.acm.org/doi/10.1145/3604915.3608818
Show More Cited By

Index Terms

NAS-CTR: Efficient Neural Architecture Search for Click-Through Rate Prediction
1. Computing methodologies
  1. Artificial intelligence
    1. Search methodologies
  2. Machine learning
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems

Recommendations

GemNN: Gating-enhanced Multi-task Neural Networks with Feature Interaction Learning for CTR Prediction
SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Deep neural network (DNN) models have been widely used for click-through rate (CTR) prediction in online advertising. The training framework typically consists of embedding layers and multi-layer perceptions (MLP). At Baidu Search Ads (a.k.a. Phoenix ...
Local Stochastic Differentiable Architecture Search for Memetic Neuroevolution Algorithms
GECCO '23 Companion: Proceedings of the Companion Conference on Genetic and Evolutionary Computation

Even the most efficient approaches to neural architecture search can be very computationally expensive, which leaves little room for inefficiencies. Unfortunately, evolutionary approaches to neural architecture search (NAS) - neuroevolution - often ...
Differentiable sampling based efficient architecture search for automatic fault diagnosis
Abstract
Intelligent diagnosis on rotating machinery has developed rapidly, but different methods have fluctuating performance and fussy design, causing poor effect in practical applications. Thus, it would be great to automatically generate the optimal ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2022

3569 pages

ISBN:9781450387323

DOI:10.1145/3477495

General Chairs:
Enrique Amigo
UNED
,
Pablo Castells
UAM and Amazon
,
Julio Gonzalo
UNED
,
Program Chairs:
Ben Carterette
Spotify
,
J. Shane Culpepper
RMIT University
,
Gabriella Kazai
Waseda University

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
Open Research Projects of Zhejiang Lab
Natural Science Foundation of Jiangsu Province
Key Research and Development Program of Jiangsu Province

Conference

SIGIR '22

Sponsor:

SIGIR

SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 11 - 15, 2022

Madrid, Spain

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
562
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)5

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Jiang SZhu GWang YYuan CHuang YBaeza-Yates RBonchi F(2024)Automatic Multi-Task Learning Framework with Neural Architecture Search in RecommendationsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671715(1290-1300)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671715
Wu CLian DGe YZhou MChen ETao D(2024)Boosting Factorization Machines via Saliency-Guided MixupIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.335491046:6(4443-4459)Online publication date: Jun-2024
https://doi.org/10.1109/TPAMI.2024.3354910
Ding RXie RHao XYang XGe KZhang XZhou JLin L(2023)Interpretable User Retention Modeling in RecommendationProceedings of the 17th ACM Conference on Recommender Systems10.1145/3604915.3608818(702-708)Online publication date: 14-Sep-2023
https://dl.acm.org/doi/10.1145/3604915.3608818
Wang FGu HLi DLu TZhang PGu NFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Towards Deeper, Lighter and Interpretable Cross Network for CTR PredictionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615089(2523-2533)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615089
Zhang YShi TFeng FWang WWang DHe XZhang YChen HDuh WHuang HKato MMothe JPoblete B(2023)Reformulating CTR Prediction: Learning Invariant Feature Interactions for RecommendationProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591755(1386-1395)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591755
Fang YMu CLiu Y(2023)AutoShape: Automatic Design of Click-Through Rate Prediction Models Using Shapley ValuePRICAI 2023: Trends in Artificial Intelligence10.1007/978-981-99-7022-3_3(29-40)Online publication date: 15-Nov-2023
https://dl.acm.org/doi/10.1007/978-981-99-7022-3_3

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten