skip to main content
10.1145/3555776.3577849acmconferencesArticle/Chapter ViewAbstractPublication PagessacConference Proceedingsconference-collections
poster

MAFD: A Federated Distillation Approach with Multi-head Attention for Recommendation Tasks

Published: 07 June 2023 Publication History

Abstract

The key challenges that recommendation systems must overcome are data isolation and privacy protection issues. Federated learning can efficiently train global models using decentralized data while preserving privacy. In real-world applications, however, it is difficult to achieve high prediction accuracy due to the heterogeneity of devices, the lack of data, and the limited generalization capacity of models. In this research, we introduce a personalized federated knowledge distillation model for a recommendation system based on a multi-head attention mechanism for recommendation systems. Specifically, we first employ federated distillation to improve the performance of student models and introduce a multi-head attention mechanism to enhance user encoding information. Next, we incorporate Wasserstein distance into the objective function of combined distillation to reduce the distribution gap between teacher and student networks and also use an adaptive learning rate technique to enhance convergence. We show that the proposed approach achieves better effectiveness and robustness through benchmarks.

References

[1]
[n. d.]. MovieLens 1M Dataset. "https://grouplens.org/datasets/movielens/1m/".
[2]
Mahmoud Abbasi, Amin Shahraki, and Amir Taherkordi. 2021. Deep learning for network traffic monitoring and analysis (NTMA): a survey. Computer Communications 170 (2021), 19--41.
[3]
Debashis Das, Laxman Sahoo, and Sujoy Datta. 2017. A survey on recommendation system. International Journal of Computer Applications 160, 7 (2017).
[4]
Sijing Duan, Deyu Zhang, Yanbo Wang, Lingxiang Li, and Yaoxue Zhang. 2019. JointRec: A deep-learning-based joint cloud video recommendation framework for mobile IoT. IEEE Internet of Things Journal 7, 3 (2019), 1655--1666.
[5]
Jialiang Han, Yun Ma, Qiaozhu Mei, and Xuanzhe Liu. 2021. Deeprec: On-device deep learning for privacy-preserving sequential recommendation in mobile commerce. In Proceedings of the Web Conference 2021. 900--911.
[6]
F Maxwell Harper and Joseph A Konstan. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) 5, 4 (2015), 1--19.
[7]
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 2, 7 (2015).
[8]
Rui Hu, Yuanxiong Guo, Hongning Li, Qingqi Pei, and Yanmin Gong. 2020. Personalized federated learning with differential privacy. IEEE Internet of Things Journal 7, 10 (2020), 9530--9539.
[9]
Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947 (2016).
[10]
Yunlong Lu, Xiaohong Huang, Yueyue Dai, Sabita Maharjan, and Yan Zhang. 2019. Differentially private asynchronous federated learning for mobile edge computing in urban informatics. IEEE Transactions on Industrial Informatics 16, 3 (2019), 2134--2143.
[11]
Paolo Massa, Kasper Souren, Martino Salvetti, and Danilo Tomasoni. 2008. Trust-let, open research on trust metrics. Scalable Computing: Practice and Experience 9, 4 (2008).
[12]
Shiva Nadi, Mohammad Hossein Saraee, Ayoub Bagheri, et al. 2011. A hybrid recommender system for dynamic web users. International Journal Multimedia and Image Processing (IJMIP) 1, 1 (2011), 3--8.
[13]
Xuanming Ni, Xinyuan Shen, and Huimin Zhao. 2022. Federated optimization via knowledge codistillation. Expert Systems with Applications 191 (2022), 116310.
[14]
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. 2019. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3967--3976.
[15]
Hyowoon Seo, Jihong Park, Seungeun Oh, Mehdi Bennis, and Seong-Lyun Kim. 2020. Federated knowledge distillation. arXiv preprint arXiv:2011.02367 (2020).
[16]
Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Deep interest evolution network for click-through rate prediction. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 5941--5948.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SAC '23: Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing
March 2023
1932 pages
ISBN:9781450395175
DOI:10.1145/3555776
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 June 2023

Check for updates

Author Tags

  1. federated learning
  2. multi-head attention
  3. wasserstein distance
  4. recommendation systems

Qualifiers

  • Poster

Conference

SAC '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25
The 40th ACM/SIGAPP Symposium on Applied Computing
March 31 - April 4, 2025
Catania , Italy

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 86
    Total Downloads
  • Downloads (Last 12 months)29
  • Downloads (Last 6 weeks)1
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media