Dongqi Cai
Yaozong Wu
Shangguang Wang
ABSTRACT
Fine-tuning pre-trained models for downstream tasks often requires private data, for which federated learning is the de-facto approach (i.e., FedNLP). However, FedNLP is prohibitively slow due to the large model sizes and the resultant high network/computation cost. Towards practical FedNLP, we identify as the key building blocks adapters, small bottleneck modules inserted at a variety of model layers. To automate adapter configuration, we propose FedAdapter 1, a framework that enhances the existing FedNLP with progressive training and sideline trial. Extensive experiments show that FedAdapter can reduce FedNLP’s model convergence delay to no more than several hours.
- [1] Dongqi Cai, Yaozong Wu, Shangguang Wang, Felix Xiaozhu Lin, and Mengwei Xu. Efficient federated learning for modern nlp. MobiCom, 2023.Google Scholar
- [2] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. ICML, 2019.Google Scholar
- [3] Bill Yuchen Lin, Chaoyang He, Zihang Ze, Hulin Wang, Yufen Hua, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. Fednlp: Benchmarking federated learning methods for natural language processing tasks. NAACL 2022, 2022.Google ScholarCross Ref
- [4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.Google ScholarDigital Library
Recommendations
Efficient Federated Learning for Modern NLP
ACM MobiCom '23: Proceedings of the 29th Annual International Conference on Mobile Computing and NetworkingTransformer-based pre-trained models have revolutionized NLP for superior performance and generality. Fine-tuning pre-trained models for downstream tasks often requires private data, for which federated learning is the de-facto approach (i.e., FedNLP)...
Federated Few-Shot Learning for Mobile NLP
ACM MobiCom '23: Proceedings of the 29th Annual International Conference on Mobile Computing and NetworkingNatural language processing (NLP) sees rich mobile applications. To support various language understanding tasks, a foundation NLP model is often fine-tuned in a federated, privacy-preserving setting (FL). This process currently relies on at least ...
Federated Machine Learning: Concept and Applications
Survey Papers and Regular PapersToday’s artificial intelligence still faces two major challenges. One is that, in most industries, data exists in the form of isolated islands. The other is the strengthening of data privacy and security. We propose a possible solution to these ...
Comments