PromptFusion: A Low-Cost Prompt-Based Task Composition for Multi-task Learning

Abstract

Prompt-tuning takes advantage of large-scale pretrained language models and achieves great performance while being more parameter-efficient. However, existing prompt-tuning methods require tuning different pretrained language models for each specific task, and fail to utilize information across different tasks, which limits their applicability in complex situations. To address above issues, we propose PromptFusion, a unique prompt-based multi-task transfer learning approach which learns knowledge from multiple tasks and incorporates for the target task at low cost. The proposed approach first learns task-specific parameters with prompts to extract information individually, then, a fusion module is designed to aggregate information. Our method is interpretable because it can explain which sources of tasks are the crucial factors to influence the model decision on the target task. We also examine a more effective way to encapsulate information by incorporating parallel adapter modules into transformer layers, and this makes a linkage between parameter-efficient transfer learning methods. We empirically evaluate our methods on the GLUE benchmark and a variety of hard NLU tasks. The results show that our approach outperforms full fine-tuning and other parameter-efficient multi-task methods.