skip to main content
10.1145/3219819.3220044acmotherconferencesArticle/Chapter ViewAbstractPublication PageskddConference Proceedingsconference-collections
research-article

Sketched Follow-The-Regularized-Leader for Online Factorization Machine

Published: 19 July 2018 Publication History

Abstract

Factorization Machine (FM) is a supervised machine learning model for feature engineering, which is widely used in many real-world applications. In this paper, we consider the case that the data samples arrive sequentially. The existing convex formulation for online FM has the strong theoretical guarantee and stable performance in practice, but the computational cost is typically expensive when the data is high-dimensional. To address this weakness, we devise a novel online learning algorithm called Sketched Follow-The-Regularizer-Leader (SFTRL). SFTRL presents the parameters of FM implicitly by maintaining low-rank matrices and updates the parameters via sketching. More specifically, we propose Generalized Frequent Directions to approximate indefinite symmetric matrices in a streaming way, making that the sum of historical gradients for FM could be estimated with tighter error bound efficiently. With mild assumptions, we prove that the regret bound of SFTRL is close to that of the standard FTRL. Experimental results show that SFTRL has better prediction quality than the state-of-the-art online FM algorithms in much lower time and space complexities.

References

[1]
Nir Ailon and Bernard Chazelle . 2006. Approximate nearest neighbors and the fast Johnson-Lindenstrauss transform STOC.
[2]
Nir Ailon and Edo Liberty . 2009. Fast dimension reduction using Rademacher series on dual BCH codes. Discrete & Computational Geometry Vol. 42, 4 (2009), 615.
[3]
Nir Ailon and Edo Liberty . 2013. An almost optimal unrestricted fast Johnson-Lindenstrauss transform. ACM Transactions on Algorithms (TALG) Vol. 9, 3 (2013), 21.
[4]
Peter L. Bartlett, Elad Hazan, and Alexander Rakhlin . 2007. Adaptive Online Gradient Descent. In NIPS.
[5]
Mathieu Blondel, Akinori Fujino, and Naonori Ueda . 2015. Convex Factorization Machines. In ECML/PKDD.
[6]
Mathieu Blondel, Akinori Fujino, Naonori Ueda, and Masakazu Ishihata . 2016. Higher-Order Factorization Machines. In NIPS.
[7]
Daniele Calandriello, Alessandro Lazaric, and Michal Valko . 2017. Second-Order Kernel Online Convex Optimization with Adaptive Sketching ICML.
[8]
Chen Cheng, Fen Xia, Tong Zhang, Irwin King, and Michael R. Lyu . 2014. Gradient boosting factorization machines. In RecSys.
[9]
Kenneth L Clarkson and David P Woodruff . 2013. Low rank approximation and regression in input sparsity time STOC.
[10]
Amey Desai, Mina Ghashami, and Jeff M Phillips . 2016. Improved practical matrix sketching with guarantees. IEEE Transactions on Knowledge and Data Engineering Vol. 28, 7 (2016), 1678--1690.
[11]
Petros Drineas, Ravi Kannan, and Michael W Mahoney . 2006 a. Fast Monte Carlo algorithms for matrices I: Approximating matrix multiplication. SIAM J. Comput. Vol. 36, 1 (2006), 132--157.
[12]
Petros Drineas, Ravi Kannan, and Michael W Mahoney . 2006 b. Fast Monte Carlo algorithms for matrices II: Computing a low-rank approximation to a matrix. SIAM Journal on computing Vol. 36, 1 (2006), 158--183.
[13]
Petros Drineas, Malik Magdon-Ismail, Michael W Mahoney, and David P Woodruff . 2012. Fast approximation of matrix coherence and statistical leverage. Journal of Machine Learning Research Vol. 13, Dec (2012), 3475--3506.
[14]
Petros Drineas, Michael W Mahoney, and S Muthukrishnan . 2008. Relative-error CUR matrix decompositions. SIAM J. Matrix Anal. Appl. Vol. 30, 2 (2008), 844--881.
[15]
John Duchi, Elad Hazan, and Yoram Singer . 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research Vol. 12, Jul (2011), 2121--2159.
[16]
Alan Frieze, Ravi Kannan, and Santosh Vempala . 2004. Fast Monte-Carlo algorithms for finding low-rank approximations. Journal of the ACM (JACM) Vol. 51, 6 (2004), 1025--1041.
[17]
Mina Ghashami, Edo Liberty, and Jeff M Phillips . 2016 a. Efficient frequent directions algorithm for sparse matrices SIGKDD.
[18]
Mina Ghashami, Edo Liberty, Jeff M Phillips, and David P Woodruff . 2016 b. Frequent directions: Simple and deterministic matrix sketching. SIAM J. Comput. Vol. 45, 5 (2016), 1762--1792.
[19]
Ming Gu and Stanley C Eisenstat . 1993. A stable and fast algorithm for updating the singular value decomposition. Technical Report YALEU/DCS/RR-966 (1993).
[20]
Weiyu Guo, Shu Wu, Liang Wang, and Tieniu Tan . 2016. Personalized ranking with pairwise Factorization Machines. Neurocomputing Vol. 214 (2016), 191--200.
[21]
Elad Hazan et almbox. . 2016. Introduction to online convex optimization. Foundations and Trends® in Optimization Vol. 2, 3--4 (2016), 157--325.
[22]
William B Johnson and Joram Lindenstrauss . 1984. Extensions of Lipschitz mappings into a Hilbert space. Contemporary mathematics Vol. 26, 189--206 (1984), 1.
[23]
Yuchin Juan, Damien Lefortier, and Olivier Chapelle . 2017. Field-aware factorization machines in a real-world online advertising system WWW Companion.
[24]
Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin . 2016. Field-aware factorization machines for CTR prediction RecSys.
[25]
Daniel M Kane and Jelani Nelson . 2014. Sparser johnson-lindenstrauss transforms. Journal of the ACM (JACM) Vol. 61, 1 (2014), 4.
[26]
Takuya Kitazawa . 2016. Incremental Factorization Machines for Persistently Cold-starting Online Item Recommendation. arXiv preprint arXiv:1607.02858 (2016). deftempurl%https://arxiv.org/abs/1607.02858 tempurl
[27]
Edo Liberty . 2013. Simple and deterministic matrix sketching. In SIGKDD.
[28]
Xiao Lin, Wenpeng Zhang, Min Zhang, Wenwu Zhu, Jian Pei, Peilin Zhao, and Junzhou Huang . 2018. Online Compact Convexified Factorization Machine. In WWW.
[29]
Chun-Ta Lu, Lifang He, Weixiang Shao, Bokai Cao, and Philip S. Yu . 2017 a. Multilinear Factorization Machines for Multi-Task Multi-View Learning WSDM.
[30]
Chun-Ta Lu, Lifang He, Weixiang Shao, Bokai Cao, and Philip S Yu . 2017 b. Multilinear factorization machines for multi-task multi-view learning WSDM.
[31]
Haipeng Luo, Alekh Agarwal, Nicolò Cesa-Bianchi, and John Langford . 2016. Efficient second order online learning by sketching NIPS.
[32]
Luo Luo, Cheng Chen, Zhihua Zhang, Wu-Jun Li, and Tong Zhang . 2017. Robust Frequent Directions with Application in Online Learning. arXiv preprint arXiv:1705.05067 (2017). deftempurl%https://arxiv.org/abs/1705.05067 tempurl
[33]
H. Brendan McMahan . 2011. Follow-the-Regularized-Leader and Mirror Descent: Equivalence Theorems and L1 Regularization. In AISTATS.
[34]
Youssef Mroueh, Etienne Marcheret, and Vaibhava Goel . 2017. Co-Occurring Directions Sketching for Approximate Matrix Multiply AISTATS.
[35]
Jelani Nelson and Huy L Nguyên . 2013. OSNAP: Faster numerical linear algebra algorithms via sparser subspace embeddings FOCS.
[36]
Trung V Nguyen, Alexandros Karatzoglou, and Linas Baltrunas . 2014. Gaussian process factorization machines for context-aware recommendations SIGIR.
[37]
Dimitris Papailiopoulos, Anastasios Kyrillidis, and Christos Boutsidis . 2014. Provable deterministic leverage score sampling. In SIGKDD.
[38]
Steffen Rendle . 2010. Factorization machines. In ICDM.
[39]
Steffen Rendle, Zeno Gantner, Christoph Freudenthaler, and Lars Schmidt-Thieme . 2011. Fast context-aware recommendations with factorization machines SIGIR.
[40]
Tamas Sarlos . 2006. Improved approximation algorithms for large matrices via random projections FOCS.
[41]
Shai Shalev-Shwartz . 2011. Online learning and online convex optimization. Foundations and Trends in Machine Learning Vol. 4, 2 (2011), 107--194.
[42]
Lloyd N Trefethen and David Bau III . 1997. Numerical linear algebra. Vol. Vol. 50. Society for Industrial and Applied Mathematics.
[43]
David P. Woodruff . 2014. Sketching as a Tool for Numerical Linear Algebra. Foundations and Trends in Theoretical Computer Science Vol. 10, 1--2 (2014), 1--157.
[44]
Makoto Yamada, Wenzhao Lian, Amit Goyal, Jianhui Chen, Kishan Wimalawarne, Suleiman A Khan, Samuel Kaski, Hiroshi Mamitsuka, and Yi Chang . 2017. Convex factorization machine for toxicogenomics prediction SIGKDD.
[45]
Qiaomin Ye, Luo Luo, and Zhihua Zhang . 2016. Frequent Direction Algorithms for Approximate Matrix Multiplication with Applications in CCA. In IJCAI.
[46]
Wenpeng Zhang, Xiao Lin, Tong Zhang, Peilin Zhao, Wenwu Zhu, Min Zhang, and Jian Pei . 2018. Compact convexified factorization machine: formulation and online algorithms. arXiv preprint arXiv:1802.01379 (2018). deftempurl%https://arxiv.org/abs/1802.01379 tempurl
[47]
Martin Zinkevich . 2003. Online Convex Programming and Generalized Infinitesimal Gradient Ascent ICML.

Cited By

View all
  • (2024)Robust Sparse Online Learning through Adversarial Sparsity Constraints2024 9th IEEE International Conference on Smart Cloud (SmartCloud)10.1109/SmartCloud62736.2024.00015(42-47)Online publication date: 10-May-2024
  • (2024)Adaptive Sparse Online Learning through Asymmetric Truncated Gradient2024 IEEE 10th International Conference on Big Data Computing Service and Machine Learning Applications (BigDataService)10.1109/BigDataService62917.2024.00013(44-51)Online publication date: 15-Jul-2024
  • (2020)IO-aware Factorization Machine for User Response Prediction2020 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN48605.2020.9207424(1-8)Online publication date: Jul-2020

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
KDD '18: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
July 2018
2925 pages
ISBN:9781450355520
DOI:10.1145/3219819
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. convex online learning
  2. factorization machine
  3. follow-the-regularized-leader
  4. sketching

Qualifiers

  • Research-article

Funding Sources

  • the NSERC Discovery Grant program
  • the Canada Research Chair program
  • National Program on Key Basic Research Project
  • National Natural Science Foundation of China
  • National Natural Science Foundation of China Major Project
  • the NSERC Strategic Grant program

Conference

KDD '18
Sponsor:

Acceptance Rates

KDD '18 Paper Acceptance Rate 107 of 983 submissions, 11%;
Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)22
  • Downloads (Last 6 weeks)2
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Robust Sparse Online Learning through Adversarial Sparsity Constraints2024 9th IEEE International Conference on Smart Cloud (SmartCloud)10.1109/SmartCloud62736.2024.00015(42-47)Online publication date: 10-May-2024
  • (2024)Adaptive Sparse Online Learning through Asymmetric Truncated Gradient2024 IEEE 10th International Conference on Big Data Computing Service and Machine Learning Applications (BigDataService)10.1109/BigDataService62917.2024.00013(44-51)Online publication date: 15-Jul-2024
  • (2020)IO-aware Factorization Machine for User Response Prediction2020 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN48605.2020.9207424(1-8)Online publication date: Jul-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media