research-article

You Say Factorization Machine, I Say Neural Network - It’s All in the Activation

Authors:

Yedid HoshenAuthors Info & Claims

RecSys '22: Proceedings of the 16th ACM Conference on Recommender Systems

Pages 389 - 398

https://doi.org/10.1145/3523227.3551499

Published: 13 September 2022 Publication History

Abstract

In recent years, many methods for machine learning on tabular data were introduced that use either factorization machines, neural networks or both. This created a great variety of methods making it non-obvious which method should be used in practice. We begin by extending the previously established theoretical connection between polynomial neural networks and factorization machines (FM) to recently introduced FM techniques. This allows us to propose a single neural-network-based framework that can switch between the deep learning and FM paradigms by a simple change of an activation function. We further show that an activation function exists which can adaptively learn to select the optimal paradigm. Another key element in our framework is its ability to learn high-dimensional embeddings by low-rank factorization. Our framework can handle numeric and categorical data as well as multiclass outputs. Extensive empirical experiments verify our analytical claims. Source code is available at https://github.com/ChenAlmagor/FiFa

Supplementary Material

MP4 File (You_Say_Factorization_Machine_I_Say_Neural_Network_-_Its_All_ in_the_Activation.mp4)

Presentation Video - FiFa: You Say Factorization Machine, I Say Neural Network - It's All in the Activation We begin by extending the connection between neural networks and factorization machines (FM) to recently introduced factorization- based methods. This allows us to propose FiFa, a single neural-network-based framework that can switch between the deep learning and FM paradigms by a simple change of an activation function. We further show that an activation function exists which can adaptively learn to select the optimal paradigm. Another key element in our framework is its ability to learn high-dimensional embeddings by low-rank factorization. Our framework can handle numeric and categorical data as well as multiclass outputs.

Download
35.47 MB

References

[1]

Sercan Ö. Arik and Tomas Pfister. 2021. TabNet: Attentive Interpretable Tabular Learning. In AAAI.

[2]

Mathieu Blondel, Masakazu Ishihata, Akinori Fujino, and Naonori Ueda. 2016. Polynomial Networks and Factorization Machines: New Insights and Efficient Training Algorithms. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016(JMLR Workshop and Conference Proceedings, Vol. 48), Maria-Florina Balcan and Kilian Q. Weinberger (Eds.). JMLR.org, 850–858. http://proceedings.mlr.press/v48/blondel16.html

[3]

Yin-Wen Chang, Cho-Jui Hsieh, Kai-Wei Chang, Michael Ringgaard, and Chih-Jen Lin. 2010. Training and Testing Low-degree Polynomial Data Mappings via Linear SVM. J. Mach. Learn. Res.(2010).

[4]

Olivier Chapelle, Eren Manavoglu, and Rómer Rosales. 2014. Simple and Scalable Response Prediction for Display Advertising. ACM Trans. Intell. Syst. Technol.(2014).

[5]

Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016. Wide & Deep Learning for Recommender Systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, DLRS@RecSys 2016, Boston, MA, USA, September 15, 2016.

Digital Library

[6]

Wei Deng, Junwei Pan, Tian Zhou, Deguang Kong, Aaron Flores, and Guang Lin. 2021. DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving. In WSDM.

[7]

Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021. Revisiting Deep Learning Models for Tabular Data. CoRR abs/2106.11959(2021). arXiv:2106.11959https://arxiv.org/abs/2106.11959

[8]

Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. In IJCAI.

[9]

Xiangnan He and Tat-Seng Chua. 2017. Neural Factorization Machines for Sparse Predictive Analytics. In SIGIR.

[10]

Dan Hendrycks and Kevin Gimpel. 2016. Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units. CoRR (2016).

[11]

Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar S. Karnin. 2020. TabTransformer: Tabular Data Modeling Using Contextual Embeddings. CoRR (2020).

[12]

Yehuda Koren. 2008. Factorization meets the neighborhood: a multifaceted collaborative filtering model. In KDD.

[13]

Zhibin Li, Jian Zhang, Yongshun Gong, Yazhou Yao, and Qiang Wu. 2020. Field-wise Learning for Multi-field Categorical Data. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/7078971350bcefbc6ec2779c9b84a9bd-Abstract.html

[14]

Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. 2018. xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. In KDD.

[15]

Roi Livni, Shai Shalev-Shwartz, and Ohad Shamir. 2014. On the Computational Efficiency of Training Neural Networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (Eds.). 855–863. https://proceedings.neurips.cc/paper/2014/hash/3a0772443a0739141292a5429b952fe6-Abstract.html

[16]

Yuanfei Luo, Hao Zhou, Wei-Wei Tu, Yuqiang Chen, Wenyuan Dai, and Qiang Yang. 2020. Network On Network for Tabular Data Classification in Real-world Applications. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, Jimmy Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (Eds.). ACM, 2317–2326. https://doi.org/10.1145/3397271.3401437

Digital Library

[17]

Junwei Pan, Jian Xu, Alfonso Lobos Ruiz, Wenliang Zhao, Shengjun Pan, Yu Sun, and Quan Lu. 2018. Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising. In WWW.

[18]

Steffen Rendle. 2010. Factorization Machines. In ICDM.

[19]

Steffen Rendle, Walid Krichene, Li Zhang, and John R. Anderson. 2020. Neural Collaborative Filtering vs. Matrix Factorization Revisited. In RecSys.

[20]

Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. 2021. SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training. CoRR (2021).

[21]

Yang Sun, Junwei Pan, Alex Zhang, and Aaron Flores. 2021. FM2: Field-matrixed Factorization Machines for Recommender Systems. In WWW. 2828–2837.

[22]

Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & Cross Network for Ad Click Predictions. In ADKDD.

[23]

Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. 2020. VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain. In NeurIPS.

[24]

Zhiyuan Zhang, Yun Liu, and Zhenjiang Zhang. 2018. Field-Aware Matrix Factorization for Recommender Systems. IEEE Access (2018).

[25]

Xinke Zhao, Wei Zeng, and Yixin He. 2021. Collaborative filtering via factorized neural networks. Appl. Soft Comput. (2021).

Cited By

Shtoff AViderman MHaramaty-Krasne NSomekh ORaviv ABan T(2024)Low Rank Field-Weighted Factorization Machines for Low Latency Item RecommendationProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688097(238-246)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688097

Index Terms

You Say Factorization Machine, I Say Neural Network - It’s All in the Activation
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Factorization methods
      2. Neural networks

Recommendations

Extracting symbolic rules from trained neural network ensembles
Special issue on Artificial intelligence advances in China

Neural network ensemble can significantly improve the generalization ability of neural network based systems. However, its comprehensibility is even worse than that of a single neural network because it comprises a collection of individual neural ...
Extracting symbolic rules from trained neural network ensembles
Artificial Intelligence Advances in China

Neural network ensemble can significantly improve the generalization ability of neural network based systems. However, its comprehensibility is even worse than that of a single neural network because it comprises a collection of individual neural ...
A recurrent neural network for computing pseudoinverse matrices

A recurrent neural network is presented for computing pseudoinverse matrices. Under the zero initial state condition, the recurrent neural network derived from a reflexive generalized inverse problem which involves two matrix equations can be used to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

RecSys '22: Proceedings of the 16th ACM Conference on Recommender Systems

September 2022

743 pages

ISBN:9781450392785

DOI:10.1145/3523227

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 September 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

RecSys '22

Sponsor:

RecSys '22: Sixteenth ACM Conference on Recommender Systems

September 18 - 23, 2022

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
966
Total Downloads

Downloads (Last 12 months)65
Downloads (Last 6 weeks)8

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Shtoff AViderman MHaramaty-Krasne NSomekh ORaviv ABan T(2024)Low Rank Field-Weighted Factorization Machines for Low Latency Item RecommendationProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688097(238-246)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688097

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten