Skip to main content

Advertisement

Springer Nature Link
Account
Menu
Find a journal Publish with us Track your research
Search
Cart
  1. Home
  2. Frontiers of Computer Science
  3. Article

A survey on LoRA of large language models

  • Review Article
  • Open access
  • Published: 14 December 2024
  • Volume 19, article number 197605, (2025)
  • Cite this article
Download PDF

You have full access to this open access article

Frontiers of Computer Science Aims and scope Submit manuscript
A survey on LoRA of large language models
Download PDF
  • Yuren Mao1,2,
  • Yuhang Ge1,
  • Yijiang Fan1,
  • Wenyi Xu1,
  • Yu Mi1,
  • Zhonghao Hu1 &
  • …
  • Yunjun Gao1,2 

  • 3150 Accesses

  • 17 Altmetric

  • Explore all metrics

Abstract

Low-Rank Adaptation (LoRA), which updates the dense neural network layers with pluggable low-rank matrices, is one of the best performed parameter efficient fine-tuning paradigms. Furthermore, it has significant advantages in cross-task generalization and privacy-preserving. Hence, LoRA has gained much attention recently, and the number of related literature demonstrates exponential growth. It is necessary to conduct a comprehensive overview of the current progress on LoRA. This survey categorizes and reviews the progress from the perspectives of (1) downstream adaptation improving variants that improve LoRA’s performance on downstream tasks; (2) cross-task generalization methods that mix multiple LoRA plugins to achieve cross-task generalization; (3) efficiency-improving methods that boost the computation-efficiency of LoRA; (4) data privacy-preserving methods that use LoRA in federated learning; (5) application. Besides, this survey also discusses the future directions in this field.

Article PDF

Download to read the full article text

Similar content being viewed by others

From Bias to Fairness: The Role of Domain-Specific Knowledge and Efficient Fine-Tuning

Chapter © 2024

Parameter-Efficient Sparse Retrievers and Rerankers Using Adapters

Chapter © 2023

Adapting LLMs to Downstream Applications

Chapter © 2024
Use our pre-submission checklist

Avoid common mistakes on your manuscript.

References

  1. Devlin J, Chang M W, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT. 2019, 4171–4186

    Google Scholar 

  2. Chowdhery A, Narang S, Devlin J, Bosma M, Mishra G, Roberts A, Barham P, Chung H W, Sutton C, Gehrmann S, Schuh P, Shi K, Tsvyashchenko S, Maynez J, Rao A, Barnes P, Tay Y, Shazeer N, Prabhakaran V, Reif E, Du N, Hutchinson B, Pope R, Bradbury J, Austin J, Isard M, Gur-Ari G, Yin P, Duke T, Levskaya A, Ghemawat S, Dev S, Michalewski H, Garcia X, Misra V, Robinson K, Fedus L, Zhou D, Ippolito D, Luan D, Lim H, Zoph B, Spiridonov A, Sepassi R, Dohan D, Agrawal S, Omernick M, Dai A M, Pillai T S, Pellat M, Lewkowycz A, Moreira E, Child R, Polozov O, Lee K, Zhou Z, Wang X, Saeta B, Diaz M, Firat O, Catasta M, Wei J, Meier-Hellstern K, Eck D, Dean J, Petrov S, Fiedel N. PaLM: scaling language modeling with pathways. The Journal of Machine Learning Research, 2023, 24(1): 240

    Google Scholar 

  3. Chen Y, Qian S, Tang H, Lai X, Liu Z, Han S, Jia J. LongLoRA: efficient fine-tuning of long-context large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024

    Google Scholar 

  4. Pan R, Liu X, Diao S, Pi R, Zhang J, Han C, Zhang T. LISA: layerwise importance sampling for memory-efficient large language model fine-tuning. 2024, arXiv preprint arXiv: 2403.17919

    Google Scholar 

  5. Ding N, Qin Y, Yang G, Wei F, Yang Z, Su Y, Hu S, Chen Y, Chan C M, Chen W, Yi J, Zhao W, Wang X, Liu Z, Zheng H T, Chen J, Liu Y, Tang J, Li J, Sun M. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 2023, 5(3): 220–235

    Article  Google Scholar 

  6. Houlsby N, Giurgiu A, Jastrzebski S, Morrone B, de Laroussilhe Q, Gesmundo A, Attariyan M, Gelly S. Parameter-efficient transfer learning for NLP. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 2790–2799

    Google Scholar 

  7. Lester B, Al-Rfou R, Constant N. The power of scale for parameter-efficient prompt tuning. In: Proceedings of 2021 Conference on Empirical Methods in Natural Language Processing. 2021, 3045–3059

    Chapter  Google Scholar 

  8. Zaken E B, Goldberg Y, Ravfogel S. BitFit: simple parameter-efficient fine-tuning for transformer-based masked language-models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022, 1–9

    Google Scholar 

  9. Hu E J, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W. LoRA: low-rank adaptation of large language models. In: Proceedings of the 10th International Conference on Learning Representations. 2022

    Google Scholar 

  10. Zhao W X, Zhou K, Li J, Tang T, Wang X, Hou Y, Min Y, Zhang B, Zhang J, Dong Z, Du Y, Yang C, Chen Y, Chen Z, Jiang J, Ren R, Li Y, Tang X, Liu Z, Liu P, Nie J Y, Wen J R. A survey of large language models. 2023, arXiv preprint arXiv: 2303.18223

    Google Scholar 

  11. Han Z, Gao C, Liu J, Zhang J, Zhang S Q. Parameter-efficient fine-tuning for large models: a comprehensive survey. 2024, arXiv preprint arXiv: 2403.14608

    Google Scholar 

  12. Malladi S, Wettig A, Yu D, Chen D, Arora S. A kernel-based view of language model fine-tuning. In: Proceedings of the 40th International Conference on Machine Learning. 2023, 23610–23641

    Google Scholar 

  13. Koubbi H, Boussard M, Hernandez L. The impact of LoRA on the emergence of clusters in transformers. 2024, arXiv preprint arXiv: 2402.15415

    Google Scholar 

  14. Jang U, Lee J D, Ryu E K. LoRA training in the NTK regime has no spurious local minima. 2024, arXiv preprint arXiv: 2402.11867

    Google Scholar 

  15. Zhu J, Greenewald K, Nadjahi K, de Ocáriz Borde H S, Gabrielsson R B, Choshen L, Ghassemi M, Yurochkin M, Solomon J. Asymmetry in low-rank adapters of foundation models. 2024, arXiv preprint arXiv: 2402.16842

    Google Scholar 

  16. Zeng Y, Lee K. The expressive power of low-rank adaptation. In: Proceedings of the 12th International Conference on Learning Representations. 2024

    Google Scholar 

  17. Lialin V, Muckatira S, Shivagunde N, Rumshisky A. ReLoRA: highrank training through low-rank updates. In: Proceedings of the 12th International Conference on Learning Representations. 2024

    Google Scholar 

  18. Jiang T, Huang S, Luo S, Zhang Z, Huang H, Wei F, Deng W, Sun F, Zhang Q, Wang D, Zhuang F. MoRA: high-rank updating for parameter-efficient fine-tuning. 2024, arXiv preprint arXiv: 2405.12130

    Google Scholar 

  19. Huh M, Cheung B, Bernstein J, Isola P, Agrawal P. Training neural networks from scratch with parallel low-rank adapters. 2024, arXiv preprint arXiv: 2402.16828

    Google Scholar 

  20. Liang Y S, Li W J. InfLoRA: interference-free low-rank adaptation for continual learning. 2024, arXiv preprint arXiv: 2404.00228

    Google Scholar 

  21. Zhao H, Ni B, Wang H, Fan J, Zhu F, Wang Y, Chen Y, Meng G, Zhang Z. Continual forgetting for pre-trained vision models. 2024, arXiv preprint arXiv: 2403.11530

    Book  Google Scholar 

  22. Ren W, Li X, Wang L, Zhao T, Qin W. Analyzing and reducing catastrophic forgetting in parameter efficient tuning. 2024, arXiv preprint arXiv: 2402.18865

    Google Scholar 

  23. Zhang H. SinkLoRA: enhanced efficiency and chat capabilities for long-context large language models. 2024, arXiv preprint arXiv: 2406.05678

    Google Scholar 

  24. Xia W, Qin C, Hazan E. Chain of LoRA: efficient fine-tuning of language models via residual learning. 2024, arXiv preprint arXiv: 2401.04151

    Google Scholar 

  25. Ren P, Shi C, Wu S, Zhang M, Ren Z, de Rijke M, Chen Z, Pei J. MELoRA: mini-ensemble low-rank adapters for parameter-efficient fine-tuning. 2024, arXiv preprint arXiv: 2402.17263

    Google Scholar 

  26. Hao Y, Cao Y, Mou L. Flora: low-rank adapters are secretly gradient compressors. 2024, arXiv preprint arXiv: 2402.03293

    Google Scholar 

  27. Zi B, Qi X, Wang L, Wang J, Wong K F, Zhang L. Delta-LoRA: fine-tuning high-rank parameters with the delta of low-rank matrices. 2023, arXiv preprint arXiv: 2309.02411

    Google Scholar 

  28. Zhang Q, Chen M, Bukharin A, He P, Cheng Y, Chen W, Zhao T. Adaptive budget allocation for parameter-efficient fine-tuning. In: Proceedings of the 11th International Conference on Learning Representations. 2023

    Google Scholar 

  29. Hu Y, Xie Y, Wang T, Chen M, Pan Z. Structure-aware low-rank adaptation for parameter-efficient fine-tuning. Mathematics, 2023, 11(20): 4317

    Article  Google Scholar 

  30. Zhang F, Li L, Chen J, Jiang Z, Wang B, Qian Y. IncreLoRA: incremental parameter allocation method for parameter-efficient fine-tuning. 2023, arXiv preprint arXiv: 2308.12043

    Google Scholar 

  31. Mao Y, Huang K, Guan C, Bao G, Mo F, Xu J. DoRA: enhancing parameter-efficient fine-tuning with dynamic rank distribution. 2024, arXiv preprint arXiv: 2405.17357

    Google Scholar 

  32. Zhang R, Qiang R, Somayajula S A, Xie P. AutoLoRA: automatically tuning matrix ranks in low-rank adaptation based on meta learning. 2024, arXiv preprint arXiv: 2403.09113

    Google Scholar 

  33. Ding N, Lv X, Wang Q, Chen Y, Zhou B, Liu Z, Sun M. Sparse low-rank adaptation of pre-trained language models. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 4133–4145

    Chapter  Google Scholar 

  34. Liu Z, Lyn J, Zhu W, Tian X, Graham Y. ALoRA: allocating low-rank adaptation for fine-tuning large language models. 2024, arXiv preprint arXiv: 2403.16187

    Google Scholar 

  35. Valipour M, Rezagholizadeh M, Kobyzev I, Ghodsi A. DyLoRA: parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023, 3274–3287

    Chapter  Google Scholar 

  36. Hayou S, Ghosh N, Yu B. The impact of initialization on LoRA finetuning dynamics. 2024, arXiv preprint arXiv: 2406.08447

    Google Scholar 

  37. Meng F, Wang Z, Zhang M. PiSSA: principal singular values and singular vectors adaptation of large language models. 2024, arXiv preprint arXiv: 2404.02948

    Google Scholar 

  38. Wang H, Xiao Z, Li Y, Wang S, Chen G, Chen Y. MiLoRA: harnessing minor singular components for parameter-efficient LLM finetuning. 2024, arXiv preprint arXiv: 2406.09044

    Google Scholar 

  39. Zhang F, Pilanci M. Riemannian preconditioned LoRA for fine-tuning foundation models. 2024, arXiv preprint arXiv: 2402.02347

    Google Scholar 

  40. Hayou S, Ghosh N, Yu B. LoRA+: efficient low rank adaptation of large models. 2024, arXiv preprint arXiv: 2402.12354

    Google Scholar 

  41. Shi S, Huang S, Song M, Li Z, Zhang Z, Huang H, Wei F, Deng W, Sun F, Zhang Q. ResLoRA: identity residual mapping in low-rank adaption. 2024, arXiv preprint arXiv: 2402.18039

    Google Scholar 

  42. Wen Z, Zhang J, Fang Y. SIBO: a simple booster for parameter-efficient fine-tuning. 2024, arXiv preprint arXiv: 2402.11896

    Google Scholar 

  43. Jin F, Liu Y, Tan Y. Derivative-free optimization for low-rank adaptation in large language models. 2024, arXiv preprint arXiv: 2403.01754

    Book  Google Scholar 

  44. Liu S Y, Wang C Y, Yin H, Molchanov P, Wang Y C F, Cheng K T, Chen M H. DoRA: weight-decomposed low-rank adaptation. 2024, arXiv preprint arXiv: 2402.09353

    Google Scholar 

  45. Qiang R, Zhang R, Xie P. BiLoRA: a bi-level optimization framework for overfitting-resilient low-rank adaptation of large pre-trained models. 2024, arXiv preprint arXiv: 2403.13037

    Google Scholar 

  46. Lin Y, Ma X, Chu X, Jin Y, Yang Z, Wang Y, Mei H. LoRA dropout as a sparsity regularizer for overfitting control. 2024, arXiv preprint arXiv: 2404.09610

    Google Scholar 

  47. Wang S, Chen L, Jiang J, Xue B, Kong L, Wu C. LoRA meets dropout under a unified framework. 2024, arXiv preprint arXiv: 2403.00812

    Book  Google Scholar 

  48. Yang A X, Robeyns M, Wang X, Aitchison L. Bayesian low-rank adaptation for large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024

    Google Scholar 

  49. Qi Z, Tan X, Shi S, Qu C, Xu Y, Qi Y. PILLOW: enhancing efficient instruction fine-tuning via prompt matching. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2023, 471–482

    Google Scholar 

  50. Zhang L, Wu J, Zhou D, Xu G. STAR: constraint LoRA with dynamic active learning for data-efficient fine-tuning of large language models. 2024, arXiv preprint arXiv: 2403.01165

    Google Scholar 

  51. Wang X, Aitchison L, Rudolph M. LoRA ensembles for large language model fine-tuning. 2023, arXiv preprint arXiv: 2310.00035

    Google Scholar 

  52. Zhao Z, Gan L, Wang G, Zhou W, Yang H, Kuang K, Wu F. LoraRetriever: input-aware LoRA retrieval and composition for mixed tasks in the wild. 2024, arXiv preprint arXiv: 2402.09997

    Google Scholar 

  53. Smith J S, Cascante-Bonilla P, Arbelle A, Kim D, Panda R, Cox D, Yang D, Kira Z, Feris R, Karlinsky L. ConStruct-VL: data-free continual structured VL concepts learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, 14994–15004

    Google Scholar 

  54. Sun Y, Li M, Cao Y, Wang K, Wang W, Zeng X, Zhao R. To be or not to be? An exploration of continuously controllable prompt engineering. 2023, arXiv preprint arXiv: 2311.09773

    Google Scholar 

  55. Zhang J, Chen S, Liu J, He J. Composing parameter-efficient modules with arithmetic operations. 2023, arXiv preprint arXiv: 2306.14870

    Google Scholar 

  56. Chitale R, Vaidya A, Kane A, Ghotkar A. Task arithmetic with LoRA for continual learning. 2023, arXiv preprint arXiv: 2311.02428

    Google Scholar 

  57. Belofsky J. Token-Level Adaptation of LoRA adapters for downstream task generalization. In: Proceedings of the 6th Artificial Intelligence and Cloud Computing Conference. 2023, 168–172

    Google Scholar 

  58. Jiang W, Lin B, Shi H, Zhang Y, Li Z, Kwok J T. Effective and parameter-efficient reusing fine-tuned models. 2023, arXiv preprint arXiv: 2310.01886

    Google Scholar 

  59. Asadi N, Beitollahi M, Khalil Y, Li Y, Zhang G, Chen X. Does combining parameter-efficient modules improve few-shot transfer accuracy? 2024, arXiv preprint arXiv: 2402.15414

    Google Scholar 

  60. Huang C, Liu Q, Lin B Y, Pang T, Du C, Lin M. LoraHub: efficient cross-task generalization via dynamic LoRA composition. 2023, arXiv preprint arXiv: 2307.13269

    Google Scholar 

  61. Yadav P, Choshen L, Raffel C, Bansal M. ComPEFT: compression for communicating parameter efficient updates via sparsification and quantization. 2023, arXiv preprint arXiv: 2311.13171

    Google Scholar 

  62. Tang A, Shen L, Luo Y, Zhan Y, Hu H, Du B, Chen Y, Tao D. Parameter-efficient multi-task model fusion with partial linearization. In: Proceedings of the 12th International Conference on Learning Representations. 2024

    Google Scholar 

  63. Shen Y, Xu Z, Wang Q, Cheng Y, Yin W, Huang L. Multimodal instruction tuning with conditional mixture of LoRA. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024, 637–648

    Chapter  Google Scholar 

  64. Buehler E L, Buehler M J. X-LoRA: mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design. APL Machine Learning, 2024, 2(2): 026119

    Article  Google Scholar 

  65. Yang S, Ali M A, Wang C L, Hu L, Wang D. MoRAL: MoE augmented LoRA for LLMs’ lifelong learning. 2024, arXiv preprint arXiv: 2402.11260

    Google Scholar 

  66. Dou S, Zhou E, Liu Y, Gao S, Zhao J, Shen W, Zhou Y, Xi Z, Wang X, Fan X, Pu S, Zhu J, Zheng R, Gui T, Zhang Q, Huang X. LoRAMoE: alleviate world knowledge forgetting in large language models via MoE-style plugin. 2023, arXiv preprint arXiv: 2312.09979

    Google Scholar 

  67. Gou Y, Liu Z, Chen K, Hong L, Xu H, Li A, Yeung D Y, Kwok J T, Zhang Y. Mixture of cluster-conditional LoRA experts for vision-language instruction tuning. 2023, arXiv preprint arXiv: 2312.12379

    Google Scholar 

  68. Liu Q, Wu X, Zhao X, Zhu Y, Xu D, Tian F, Zheng Y. When MOE meets LLMs: parameter efficient fine-tuning for multi-task medical applications. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024, 1104–1114

    Chapter  Google Scholar 

  69. Feng W, Hao C, Zhang Y, Han Y, Wang H. Mixture-of-LoRAs: an efficient multitask tuning method for large language models. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 11371–11380

    Google Scholar 

  70. Wang Y, Lin Y, Zeng X, Zhang G. MultiLoRA: democratizing LoRA for better multi-task learning. 2023, arXiv preprint arXiv: 2311.11501

    Google Scholar 

  71. Yang Y, Jiang P T, Hou Q, Zhang H, Chen J, Li B. Multi-task dense prediction via mixture of low-rank experts. 2024, arXiv preprint arXiv: 2403.17749

    Book  Google Scholar 

  72. Agiza A, Neseem M, Reda S. MTLoRA: low-rank adaptation approach for efficient multi-task learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024, 16196–16205

    Google Scholar 

  73. Gao C, Chen K, Rao J, Sun B, Liu R, Peng D, Zhang Y, Guo X, Yang J, Subrahmanian V S. Higher layers need more LoRA experts. 2024, rXiv preprint arXiv: 2402.08562

    Google Scholar 

  74. Chen S, Jie Z, Ma L. LLaVA-MoLE: sparse mixture of LoRA experts for mitigating data conflicts in instruction finetuning MLLMs. 2024, arXiv preprint arXiv: 2401.16160

    Google Scholar 

  75. Zhu Y, Wichers N, Lin C C, Wang X, Chen T, Shu L, Lu H, Liu C, Luo L, Chen J, Meng L. SiRA: sparse mixture of low rank adaptation. 2023, arXiv preprint arXiv: 2311.09179

    Google Scholar 

  76. Chen Z, Wang Z, Wang Z, Liu H, Yin Z, Liu S, Sheng L, Ouyang W, Qiao Y, Shao J. Octavius: mitigating task interference in MLLMs via MoE. 2023, arXiv preprint arXiv: 2311.02684

    Google Scholar 

  77. Wen Y, Chaudhuri S. Batched low-rank adaptation of foundation models. In: Proceedings of the Twelfth International Conference on Learning Representations. 2024

    Google Scholar 

  78. Wu T, Wang J, Zhao Z, Wong N. Mixture-of-Subspaces in Low-Rank Adaptation. 2024, arXiv preprint arXiv:2406.11909

    Book  Google Scholar 

  79. Wu Y, Xiang Y, Huo S, Gong Y, Liang P. LoRA-SP: streamlined partial parameter adaptation for resource efficient fine-tuning of large language models. In: Proceedings of the 3rd International Conference on Algorithms, Microchips, and Network Applications. 2024, 131711Z

    Google Scholar 

  80. Zhang L, Zhang L, Shi S, Chu X, Li B. LoRA-FA: memory-efficient low-rank adaptation for large language models fine-tuning. 2023, arXiv preprint arXiv: 2308.03303

    Google Scholar 

  81. Liu Z, Kundu S, Li A, Wan J, Jiang L, Beerel P A. AFLoRA: adaptive freezing of low rank adaptation in parameter efficient fine-tuning of large models. 2024, arXiv preprint arXiv: 2403.13269

    Google Scholar 

  82. Woo S, Park B, Kim B, Jo M, Kwon S, Jeon D, Lee D. DropBP: accelerating fine-tuning of large language models by dropping backward propagation. 2024, arXiv preprint arXiv: 2402.17812

    Google Scholar 

  83. Bałazy K, Banaei M, Aberer K, Tabor J. LoRA-XS: low-rank adaptation with extremely small number of parameters. 2024, arXiv preprint arXiv: 2405.17604

    Google Scholar 

  84. Zhou H, Lu X, Xu W, Zhu C, Zhao T, Yang M. LoRA-drop: efficient LoRA parameter pruning based on output evaluation. 2024, arXiv preprint arXiv: 2402.07721

    Google Scholar 

  85. Zhang M, Chen H, Shen C, Yang Z, Ou L, Yu X, Zhuang B. LoRAPrune: structured pruning meets low-rank parameter-efficient fine-tuning. In: Proceedings of the Findings of the Association for Computational Linguistics. 2024, 3013–3026

    Google Scholar 

  86. Chen T, Ding T, Yadav B, Zharkov I, Liang L. LoRAShear: efficient large language model structured pruning and knowledge recovery. 2023, arXiv preprint arXiv: 2310.18356

    Google Scholar 

  87. Zhu Y, Yang X, Wu Y, Zhang W. Parameter-efficient fine-tuning with layer pruning on free-text sequence-to-sequence modeling. 2023, arXiv preprint arXiv: 2305.08285

    Google Scholar 

  88. Kopiczko D J, Blankevoort T, Asano Y M. VeRA: vector-based random matrix adaptation. In: Proceedings of the 12th International Conference on Learning Representations. 2024

    Google Scholar 

  89. Li Y, Han S, Ji S. VB-LoRA: extreme parameter efficient fine-tuning with vector banks. 2024, arXiv preprint arXiv: 2405.15179

    Google Scholar 

  90. Gao Z, Wang Q, Chen A, Liu Z, Wu B, Chen L, Li J. Parameter-efficient fine-tuning with discrete Fourier transform. 2024, arXiv preprint arXiv: 2405.03003

    Google Scholar 

  91. Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L. QLORA: efficient finetuning of quantized LLMs. In: Proceedings of the 37th International Conference on Neural Information Processing Systems.

  92. Xu Y, Xie L, Gu X, Chen X, Chang H, Zhang H, Chen Z, Zhang X, Tian Q. QA-LoRA: quantization-aware low-rank adaptation of large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024

    Google Scholar 

  93. Li Y, Yu Y, Liang C, He P, Karampatziakis N, Chen W, Zhao T. LoftQ: LoRA-fine-tuning-aware quantization for large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024

    Google Scholar 

  94. Liao B, Herold C, Khadivi S, Monz C. ApiQ: finetuning of 2-bit quantized large language model. 2024, arXiv preprint arXiv: 2402.05147

    Google Scholar 

  95. Jeon H, Kim Y, Kim J J. L4Q: parameter efficient quantization-aware training on large language models via LoRA-wise LSQ. 2024, arXiv preprint arXiv: 2402.04902

    Google Scholar 

  96. Ye Z, Li D, Tian J, Lan T, Zuo J, Duan L, Lu H, Jiang Y, Sha J, Zhang K, Tang M. ASPEN: high-throughput LoRA fine-tuning of large language models with a single GPU. 2023, arXiv preprint arXiv: 2312.02515

    Google Scholar 

  97. Chen L, Ye Z, Wu Y, Zhuo D, Ceze L, Krishnamurthy A. Punica: multi-tenant LoRA serving. In: Proceedings of the Seventh Annual Conference on Machine Learning and Systems. 2024, 1–13

    Google Scholar 

  98. Sheng Y, Cao S, Li D, Hooper C, Lee N, Yang S, Chou C, Zhu B, Zheng L, Keutzer K, Gonzalez J E, Stoica I. S-LoRA: serving thousands of concurrent LoRA adapters. 2023, arXiv preprint arXiv: 2311.03285

    Google Scholar 

  99. Li S, Lu H, Wu T, Yu M, Weng Q, Chen X, Shan Y, Yuan B, Wang W. CaraServe: CPU-assisted and rank-aware LoRA serving for generative LLM inference. 2024, arXiv preprint arXiv: 2401.11240

    Google Scholar 

  100. Babakniya S, Elkordy A R, Ezzeldin Y H, Liu Q, Song K B, El-Khamy M, Avestimehr S. SLoRA: federated parameter efficient fine-tuning of language models. 2023, arXiv preprint arXiv: 2308.06522

    Google Scholar 

  101. Yan Y, Tang S, Shi Z, Yang Q. FeDeRA: efficient fine-tuning of language models in federated learning leveraging weight decomposition. 2024, arXiv preprint arXiv: 2404.18848

    Google Scholar 

  102. Sun Y, Li Z, Li Y, Ding B. Improving LoRA in privacy-preserving federated learning. In: Proceedings of the 12th International Conference on Learning Representations. 2024

    Google Scholar 

  103. Wu P, Li K, Wang T, Wang F. FedMS: federated learning with mixture of sparsely activated foundations models. 2023, arXiv preprint arXiv: 2312.15926

    Google Scholar 

  104. Bai J, Chen D, Qian B, Yao L, Li Y. Federated fine-tuning of large language models under heterogeneous language tasks and client resources. 2024, arXiv preprint arXiv: 2402.11505

    Google Scholar 

  105. Cho Y J, Liu L, Xu Z, Fahrezi A, Barnes M, Joshi G. Heterogeneous LoRA for federated fine-tuning of on-device foundation models. In: Proceedings of the International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS. 2023

    Google Scholar 

  106. Yi L, Yu H, Wang G, Liu X, Li X. pFedLoRA: model-heterogeneous personalized federated learning with LoRA tuning. 2023, arXiv preprint arXiv: 2310.13283

    Google Scholar 

  107. Huang W, Wang Y, Cheng A, Zhou A, Yu C, Wang L. A fast, performant, secure distributed training framework for large language model. 2024, arXiv preprint arXiv: 2401.09796

    Google Scholar 

  108. Wang Y, Lin Y, Zeng X, Zhang G. PrivateLoRA for efficient privacy preserving LLM. 2023, arXiv preprint arXiv: 2311.14030

    Google Scholar 

  109. Zhang Y, Wang M, Wu Y, Tiwari P, Li Q, Wang B, Qin J. DialogueLLM: context and emotion knowledge-tuned large language models for emotion recognition in conversations. 2024, arXiv preprint arXiv: 2310.11374

    Google Scholar 

  110. Li Z, Li X, Liu Y, Xie H, Li J, Wang F L, Li Q, Zhong X. Label supervised LLaMA finetuning. 2023, arXiv preprint arXiv: 2310.01208

    Google Scholar 

  111. Bornheim T, Grieger N, Blaneck P G, Bialonski S. Speaker attribution in German parliamentary debates with QLoRA-adapted large language models. 2024, arXiv preprint arXiv: 2309.09902

    Book  Google Scholar 

  112. Xue L, Zhang D, Dong Y, Tang J. AutoRE: document-level relation extraction with large language models. 2024, arXiv preprint arXiv: 2403.14888

    Google Scholar 

  113. Alves D M, Guerreiro N M, Alves J, Pombal J, Rei R, de Souza J G C, Colombo P, Martins A F T. Steering large language models for machine translation with finetuning and in-context learning. In: Proceedings of the Findings of the Association for Computational Linguistics. 2023, 11127–11148

    Google Scholar 

  114. Zheng J, Hong H, Wang X, Su J, Liang Y, Wu S. Fine-tuning large language models for domain-specific machine translation. 2024, arXiv preprint arXiv: 2402.15061

    Google Scholar 

  115. Mujadia V, Urlana A, Bhaskar Y, Pavani P A, Shravya K, Krishnamurthy P, Sharma D M. Assessing translation capabilities of large language models involving English and Indian languages. 2023, arXiv preprint arXiv: 2311.09216

    Google Scholar 

  116. Zhang Y, Wang J, Yu L C, Xu D, Zhang X. Personalized LoRA for human-centered text understanding. In: Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence. 2024, 19588–19596

    Google Scholar 

  117. Liu Y, An C, Qiu X. Y-tuning: an efficient tuning paradigm for large-scale pre-trained models via label representation learning. Frontiers of Computer Science, 2024, 18(4): 184320

    Article  Google Scholar 

  118. Liu S, Keung J, Yang Z, Liu F, Zhou Q, Liao Y. Delving into parameter-efficient fine-tuning in code change learning: an empirical study. In: Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 2024, 465–476

    Google Scholar 

  119. Guo Y, Gao X, Jiang B. An empirical study on JIT defect prediction based on BERT-style model. 2024, arXiv preprint arXiv: 2403.11158

    Google Scholar 

  120. Ayupov S, Chirkova N. Parameter-efficient finetuning of transformers for source code. 2022, arXiv preprint arXiv: 2212.05901

    Google Scholar 

  121. Silva A, Fang S, Monperrus M. RepairLLaMA: efficient representations and fine-tuned adapters for program repair. 2023, arXiv preprint arXiv: 2312.15698

    Google Scholar 

  122. Roberson R, Kaki G, Trivedi A. Analyzing the effectiveness of large language models on text-to-SQL synthesis. 2024, arXiv preprint arXiv: 2401.12379

    Google Scholar 

  123. Pan J, Sadé A, Kim J, Soriano E, Sole G, Flamant S. SteloCoder: a decoder-only LLM for multi-language to python code translation. 2023, arXiv preprint arXiv: 2310.15539

    Google Scholar 

  124. Sidahmed H, Phatale S, Hutcheson A, Lin Z, Chen Z, Yu Z, Jin J, Komarytsia R, Ahlheim C, Zhu Y, Chaudhary S, Li B, Ganesh S, Byrne B, Hoffmann J, Mansoor H, Li W, Rastogi A, Dixon L. PERL: parameter efficient reinforcement learning from human feedback. 2024, arXiv preprint arXiv: 2403.10704

    Google Scholar 

  125. Santacroce M, Lu Y, Yu H, Li Y, Shen Y. Efficient RLHF: reducing the memory usage of PPO. 2023, arXiv preprint arXiv: 2309.00754

    Google Scholar 

  126. Sun S, Gupta D, Iyyer M. Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF. 2023, arXiv preprint arXiv: 2309.09055

    Google Scholar 

  127. Quan S. DMoERM: recipes of mixture-of-experts for effective reward modeling. 2024, arXiv preprint arXiv: 2403.01197

    Google Scholar 

  128. Zhang S, Chen Z, Chen S, Shen Y, Sun Z, Gan C. Improving reinforcement learning from human feedback with efficient reward model ensemble. 2024, arXiv preprint arXiv: 2401.16635

    Google Scholar 

  129. Zhai Y, Zhang H, Lei Y, Yu Y, Xu K, Feng D, Ding B, Wang H. Uncertainty-penalized reinforcement learning from human feedback with diverse reward LoRA ensembles. 2023, arXiv preprint arXiv: 2401.00243

    Google Scholar 

  130. Yang A X, Robeyns M, Coste T, Shi Z, Wang J, Bou-Ammar H, Aitchison L. Bayesian reward models for LLM alignment. 2024, arXiv preprint arXiv: 2402.13210

    Google Scholar 

  131. Daxberger E, Kristiadi A, Immer A, Eschenhagen R, Bauer M, Hennig P. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems. 2021

    Google Scholar 

  132. Tran H, Yang Z, Yao Z, Yu H. BioInstruct: instruction tuning of large language models for biomedical natural language processing. 2023, arXiv preprint arXiv: 2310.19975

    Google Scholar 

  133. Gema A P, Minervini P, Daines L, Hope T, Alex B. Parameter-efficient fine-tuning of LLaMA for the clinical domain. 2023, arXiv preprint arXiv: 2307.03042

    Google Scholar 

  134. Toma A, Lawler P R, Ba J, Krishnan R G, Rubin B B, Wang B. Clinical camel: an open-source expert-level medical language model with dialogue-based knowledge encoding. 2023, arXiv preprint arXiv: 2305.12031

    Google Scholar 

  135. Suri K, Mishra P, Saha S, Singh A. Suryakiran at MEDIQA-Sum 2023: leveraging LoRA for clinical dialogue summarization. In: Proceedings of the Working Notes of the Conference and Labs of the Evaluation Forum. 2023, 1720–1735

    Google Scholar 

  136. Ji Y, Yu Z, Wang Y. Assertion detection large language model in-context learning LoRA fine-tuning. 2024, arXiv preprint arXiv: 2401.17602

    Google Scholar 

  137. Wang R, Duan Y, Lam C, Chen J, Xu J, Chen H, Liu X, Pang P C I, Tan T. IvyGPT: InteractiVe Chinese pathway language model in medical domain. In: Proceedings of the 3rd CAAI International Conference on Artificial Intelligence. 2024, 378–382

    Google Scholar 

  138. Bhatti A, Parmar S, Lee S. SM70: a large language model for medical devices. 2023, arXiv preprint arXiv: 2312.06974

    Google Scholar 

  139. Konstantinidis T, Iacovides G, Xu M, Constantinides T G, Mandic D. FinLlama: financial sentiment classification for algorithmic trading applications. 2024, arXiv preprint arXiv: 2403.12285

    Google Scholar 

  140. Pavlyshenko B M. Financial news analytics using fine-tuned llama 2 GPT model. 2023, arXiv preprint arXiv: 2308.13032

    Google Scholar 

  141. Liu X Y, Wang G, Yang H, Zha D. FinGPT: democratizing internetscale data for financial large language models. 2023, arXiv preprint arXiv: 2307.10485

    Google Scholar 

  142. Li J, Lei Y, Bian Y, Cheng D, Ding Z, Jiang C. RA-CFGPT: Chinese financial assistant with retrieval-augmented large language model. Frontiers of Computer Science, 2024, 18(5): 185350

    Article  Google Scholar 

  143. Zhou X, Sun Z, Li G. DB-GPT: large language model meets database. Data Science and Engineering, 2024, 9(1): 102–111

    Article  Google Scholar 

  144. Li S. DiffStyler: diffusion-based localized image style transfer. 2024, arXiv preprint arXiv: 2403.18461

    Google Scholar 

  145. Frenkel Y, Vinker Y, Shamir A, Cohen-Or D. Implicit style-content separation using B-LoRA. 2024, arXiv preprint arXiv: 2403.14572

    Google Scholar 

  146. Liu Y, Yu C, Shang L, He Y, Wu Z, Wang X, Xu C, Xie H, Wang W, Zhao Y, Zhu L, Cheng C, Chen W, Yao Y, Zhou W, Xu J, Wang Q, Chen Y, Xie X, Sun B. FaceChain: a playground for human-centric artificial intelligence generated content. 2023, arXiv preprint arXiv: 2308.14256

    Google Scholar 

  147. Liao Q, Xia G, Wang Z. Calliffusion: Chinese calligraphy generation and style transfer with diffusion modeling. 2023, arXiv preprint arXiv: 2305.19124

    Google Scholar 

  148. Shrestha S, Sripada V S S, Venkataramanan A. Style transfer to Calvin and Hobbes comics using stable diffusion. 2023, arXiv preprint arXiv: 2312.03993

    Google Scholar 

  149. Li L, Zeng H, Yang C, Jia H, Xu D. Block-wise LoRA: revisiting finegrained LoRA for effective personalization and stylization in text-to-image generation. 2024, arXiv preprint arXiv: 2403.07500

    Google Scholar 

  150. Kong Z, Zhang Y, Yang T, Wang T, Zhang K, Wu B, Chen G, Liu W, Luo W. OMG: occlusion-friendly personalized multi-concept generation in diffusion models. 2024, arXiv preprint arXiv: 2403.10983

    Google Scholar 

  151. Shi J, Hua H. Space narrative: generating images and 3D scenes of Chinese garden from text using deep learning. In: Proceedings of the xArch-Creativity in the Age of Digital Reproduction Symposium. 2024, 236–243

    Chapter  Google Scholar 

  152. Jin Z, Song Z. Generating coherent comic with rich story using ChatGPT and stable diffusion. 2023, arXiv preprint arXiv: 2305.11067

    Google Scholar 

  153. Wang H, Xiang X, Fan Y, Xue J H. Customizing 360-degree panoramas through text-to-image diffusion models. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024, 4921–4931

    Google Scholar 

  154. Guo J, Xu X, Pu Y, Ni Z, Wang C, Vasu M, Song S, Huang G, Shi H. Smooth diffusion: crafting smooth latent spaces in diffusion models. 2023, arXiv preprint arXiv: 2312.04410

    Google Scholar 

  155. Cheng J, Xie P, Xia X, Li J, Wu J, Ren Y, Li H, Xiao X, Zheng M, Fu L. ResAdapter: domain consistent resolution adapter for diffusion models. 2024, arXiv preprint arXiv: 2403.02084

    Google Scholar 

  156. Smith J S, Hsu Y C, Kira Z, Shen Y, Jin H. Continual diffusion with STAMINA: STack-and-mask INcremental adapters. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, 1744–1754

    Google Scholar 

  157. Sun J, Fu D, Hu Y, Wang S, Rassin R, Juan D C, Alon D, Herrmann C, van Steenkiste S, Krishna R, Rashtchian C. Dreamsync: aligning text-to-image generation with image understanding feedback. In: Proceedings of the Synthetic Data for Computer Vision Workshop@CVPR 2024. 2023

    Google Scholar 

  158. Wang Z, Wang X, Xie L, Qi Z, Shan Y, Wang W, Luo P. StyleAdapter: a single-pass LoRA-free model for stylized image generation. 2023, arXiv preprint arXiv: 2309.01770

    Google Scholar 

  159. Gu Y, Wang X, Wu J Z, Shi Y, Chen Y, Fan Z, Xiao W, Zhao R, Chang S, Wu W, Ge Y, Shan Y, Shou M Z. Mix-of-show: decentralized low-rank adaptation for multi-concept customization of diffusion models. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023

    Google Scholar 

  160. Luo S, Tan Y, Patil S, Gu D, von Platen P, Passos A, Huang L, Li J, Zhao H. LCM-LoRA: a universal stable-diffusion acceleration module. 2023, arXiv preprint arXiv: 2311.05556

    Google Scholar 

  161. Golnari P A. LoRA-enhanced distillation on guided diffusion models. 2023, arXiv preprint arXiv: 2312.06899

    Google Scholar 

  162. Ren Y, Zhou Y, Yang J, Shi J, Liu D, Liu F, Kwon M, Shrivastava A. Customize-A-video: one-shot motion customization of text-to-video diffusion models. 2024, arXiv preprint arXiv: 2402.14780

    Google Scholar 

  163. Deng Y, Wang R, Zhang Y, Tai Y W, Tang C K. DragVideo: interactive drag-style video editing. 2023, arXiv preprint arXiv: 2312.02216

    Google Scholar 

  164. Yang S, Zhou Y, Liu Z, Loy C C. Rerender A video: zero-shot text-guided video-to-video translation. In: Proceedings of the SIGGRAPH Asia 2023 Conference Papers. 2023, 95

    Google Scholar 

  165. Khandelwal A. InFusion: inject and attention fusion for multi concept zero-shot text-based video editing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2023, 3009–3018

    Google Scholar 

  166. Blattmann A, Dockhorn T, Kulal S, Mendelevitch D, Kilian M, Lorenz D, Levi Y, English Z, Voleti V, Letts A, Jampani V, Rombach R. Stable video diffusion: scaling latent video diffusion models to large datasets. 2023, arXiv preprint arXiv: 2311.15127

    Google Scholar 

  167. Guo Y, Yang C, Rao A, Liang Z, Wang Y, Qiao Y, Agrawala M, Lin D, Dai B. AnimateDiff: animate your personalized text-to-image diffusion models without specific tuning. In: Proceedings of the 12th International Conference on Learning Representations. 2024

    Google Scholar 

  168. Huang T, Zeng Y, Zhang Z, Xu W, Xu H, Xu S, Lau R W H, Zuo W. DreamControl: control-based text-to-3D generation with 3D self-prior. 2023, arXiv preprint arXiv: 2312.06439

    Google Scholar 

  169. Ma Y, Fan Y, Ji J, Wang H, Sun X, Jiang G, Shu A, Ji R. X-dreamer: creating high-quality 3D content by bridging the domain gap between text-to-2D and text-to-3D generation. 2023, arXiv preprint arXiv: 2312.00085

    Google Scholar 

  170. Yu K, Liu J, Feng M, Cui M, Xie X. Boosting3D: high-fidelity image-to-3D by boosting 2D diffusion prior to 3D prior with progressive learning. 2023, arXiv preprint arXiv: 2311.13617

    Google Scholar 

  171. Yoo S, Kim K, Kim V G, Sung M. As-plausible-as-possible: plausibility-aware mesh deformation using 2D diffusion priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, 4315–4324

    Google Scholar 

  172. Zhang Y, Xu Q, Zhang L. DragTex: generative point-based texture editing on 3D mesh. 2024, arXiv preprint arXiv: 2403.02217

    Google Scholar 

  173. Ding H, Gao J, Yuan Y, Wang Q. SamLP: a customized segment anything model for license plate detection. 2024, arXiv preprint arXiv: 2401.06374

    Google Scholar 

  174. Ye Z, Lovell L, Faramarzi A, Ninic J. SAM-based instance segmentation models for the automation of structural damage detection. 2024, arXiv preprint arXiv: 2401.15266

    Google Scholar 

  175. Na S, Guo Y, Jiang F, Ma H, Huang J. Segment any cell: a SAM-based auto-prompting fine-tuning framework for nuclei segmentation. 2024, arXiv preprint arXiv: 2401.13220

    Google Scholar 

  176. Chen X, Wang C, Ning H, Li S, Shen M. SAM-OCTA: prompting segment-anything for OCTA image segmentation. 2023, arXiv preprint arXiv: 2310.07183

    Google Scholar 

  177. Feng W, Zhu L, Yu L. Cheap lunch for medical image segmentation by fine-tuning SAM on few exemplars. 2023, arXiv preprint arXiv: 2308.14133

    Google Scholar 

  178. Zhang K, Liu D. Customized segment anything model for medical image segmentation. 2023, arXiv preprint arXiv: 2304.13785

    Google Scholar 

  179. Wang A, Islam M, Xu M, Zhang Y, Ren H. SAM meets robotic surgery: an empirical study on generalization, robustness and adaptation. In: Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. 2023, 234–244

    Google Scholar 

  180. Lin L, Fan H, Zhang Z, Wang Y, Xu Y, Ling H. Tracking meets LoRA: faster training, larger model, stronger performance. 2024, arXiv preprint arXiv: 2403.05231

    Google Scholar 

  181. Kong C, Li H, Wang S. Enhancing general face forgery detection via vision transformer with low-rank adaptation. In: Proceedings of the 6th International Conference on Multimedia Information Processing and Retrieval. 2023, 102–107

    Google Scholar 

  182. Chen Z, Huang H, Andrusenko A, Hrinchuk O, Puvvada K C, Li J, Ghosh S, Balam J, Ginsburg B. SALM: speech-augmented language model with in-context learning for speech recognition and translation. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2024, 13521–13525

    Google Scholar 

  183. Dong X, Zhang P, Zang Y, Cao Y, Wang B, Ouyang L, Wei X, Zhang S, Duan H, Cao M, Zhang W, Li Y, Yan H, Gao Y, Zhang X, Li W, Li J, Chen K, He C, Zhang X, Qiao Y, Lin D, Wang J. InternLMXComposer2: mastering free-form text-image composition and comprehension in vision-language large model. 2024, arXiv preprint arXiv: 2401.16420

    Google Scholar 

  184. Ye Q, Xu H, Xu G, Ye J, Yan M, Zhou Y, Wang J, Hu A, Shi P, Shi Y, Li C, Xu Y, Chen H, Tian J, Qian Q, Zhang J, Huang F, Zhou J. mPLUG-Owl: modularization empowers large language models with multimodality. 2023, arXiv preprint arXiv: 2304.14178

    Google Scholar 

  185. Lee B K, Park B, Kim C W, Ro Y M. CoLLaVO: crayon large language and vision mOdel. 2024, arXiv preprint arXiv: 2402.11248

    Google Scholar 

  186. Yeo J H, Han S, Kim M, Ro Y M. Where visual speech meets language: VSP-LLM framework for efficient and context-aware visual speech processing. 2024, arXiv preprint arXiv: 2402.15151

    Google Scholar 

  187. Liu Z, Li S, Luo Y, Fei H, Cao Y, Kawaguchi K, Wang X, Chua T S. MolCA: molecular graph-language modeling with cross-modal projector and uni-modal adapter. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 15623–15638

    Chapter  Google Scholar 

  188. Ren Y, Chen Y, Liu S, Wang B, Yu H, Cui Z. TPLLM: a traffic prediction framework based on pretrained large language models. 2024, arXiv preprint arXiv: 2403.02221

    Google Scholar 

  189. Aghajanyan A, Gupta S, Zettlemoyer L. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2021, 7319–7328

    Google Scholar 

  190. Fomenko V, Yu H, Lee J, Hsieh S, Chen W. A note on LoRA. 2024, arXiv preprint arXiv: 2404.05086

    Google Scholar 

  191. Bershatsky D, Cherniuk D, Daulbaev T, Mikhalev A, Oseledets I. LoTR: low tensor rank weight adaptation. 2024, arXiv preprint arXiv: 2402.01376

    Google Scholar 

  192. Edalati A, Tahaei M, Kobyzev I, Nia V P, Clark J J, Rezagholizadeh M. KronA: parameter efficient tuning with kronecker adapter. 2022, arXiv preprint arXiv: 2212.10650

    Google Scholar 

  193. He X, Li C, Zhang P, Yang J, Wang X E. Parameter-efficient model adaptation for vision transformers. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 817–825

    Google Scholar 

  194. Zhao Z, Gan L, Wang G, Hu Y, Shen T, Yang H, Kuang K, Wu F. Retrieval-augmented mixture of lora experts for uploadable machine learning. 2024, arXiv preprint arXiv:2406.16989.

    Google Scholar 

  195. Mahabadi R K, Henderson J, Ruder S. COMPACTER: efficient low-rank hypercomplex adapter layers. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 79

    Google Scholar 

  196. Liao B, Meng Y, Monz C. Parameter-efficient fine-tuning without introducing new latency. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 4242–4260

    Google Scholar 

  197. Hendrycks D, Burns C, Basart S, Zou A, Mazeika M, Song D, Steinhardt J. Measuring massive multitask language understanding. In: Proceedings of the 9th International Conference on Learning Representations. 2021

    Google Scholar 

  198. He J, Zhou C, Ma X, Berg-Kirkpatrick T, Neubig G. Towards a unified view of parameter-efficient transfer learning. In: Proceedings of the 10th International Conference on Learning Representations. 2022

    Google Scholar 

  199. Geshkovski B, Letrouit C, Polyanskiy Y, Rigollet P. A mathematical perspective on transformers. 2023, arXiv preprint arXiv: 2312.10794

    Google Scholar 

  200. Geshkovski B, Letrouit C, Polyanskiy Y, Rigollet P. The emergence of clusters in self-attention dynamics. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023

    Google Scholar 

  201. Sander M E, Ablin P, Blondel M, Peyré G. Sinkformers: transformers with doubly stochastic attention. In: Proceedings of the 25th International Conference on Artificial Intelligence and Statistics. 2022, 3515–3530

    Google Scholar 

  202. Jacot A, Gabriel F, Hongler C. Neural tangent kernel: convergence and generalization in neural networks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 8580–8589

    Google Scholar 

  203. Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, Bashlykov N, Batra S, Bhargava P, Bhosale S, Bikel D, Blecher L, Canton Ferrer C, Chen M, Cucurull G, Esiobu D, Fernandes J, Fu J, Fu W, Fuller B, Gao C, Goswami V, Goyal N, Hartshorn A, Hosseini S, Hou R, Inan H, Kardas M, Kerkez V, Khabsa M, Kloumann I, Korenev A, Koura P S, Lachaux M A, Lavril T, Lee J, Liskovich D, Lu Y, Mao Y, Martinet X, Mihaylov T, Mishra P, Molybog I, Nie Y, Poulton A, Reizenstein J, Rungta R, Saladi K, Schelten A, Silva R, Smith E M, Subramanian R, Tan X E, Tang B, Taylor R, Williams A, Kuan J X, Xu P, Yan Z, Zarov I, Zhang Y, Fan A, Kambadur M, Narang S, Rodriguez A, Stojnic R, Edunov S, Scialom T. Llama 2: open foundation and fine-tuned chat models. 2023, arXiv preprint arXiv: 2307.09288

    Google Scholar 

  204. Chang Y, Chang Y, Wu Y. Bias-Aware Low-Rank Adaptation: Mitigating Catastrophic Inheritance of Large Language Models. 2024, arXiv preprint arXiv:2408.04556

    Google Scholar 

  205. Zhao J, Zhang Z, Chen B, Wang Z, Anandkumar A, Tian Y. Galore: memory-efficient LLM training by gradient low-rank projection. 2024, arXiv preprint arXiv: 2403.03507

    Google Scholar 

  206. Biderman D, Ortiz J G, Portes J, Paul M, Greengard P, Jennings C, King D, Havens S, Chiley V, Frankle J, Blakeney C, Cunningham J P. LoRA learns less and forgets less. 2024, arXiv preprint arXiv: 2405.09673

    Google Scholar 

  207. Han A, Li J, Huang W, Hong M, Takeda A, Jawanpuria P, Mishra B. SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining. 2024, arXiv preprint arXiv: 2406.02214

    Google Scholar 

  208. Sui Y, Yin M, Gong Y, Xiao J, Phan H, Yuan B. ELRT: efficient low-rank training for compact convolutional neural networks. 2024, arXiv preprint arXiv: 2401.10341

    Google Scholar 

  209. Meng X, Dai D, Luo W, Yang Z, Wu S, Wang X, Wang P, Dong Q, Chen L, Sui Z. PeriodicLoRA: breaking the low-rank bottleneck in LoRA optimization. 2024, arXiv preprint arXiv: 2402.16141

    Google Scholar 

  210. Frank M, Wolfe P. An algorithm for quadratic programming. Naval Research Logistics Quarterly, 1956, 3(1–2): 95–110

    Article  MathSciNet  Google Scholar 

  211. Rajabzadeh H, Valipour M, Zhu T, Tahaei M, Kwon HJ, Ghodsi A, Chen B, Rezagholizadeh M. Qdylora: Quantized dynamic low-rank adaptation for efficient large language model tuning. 2024, arXiv preprint arXiv:2402.10462

    Google Scholar 

  212. Elsken T, Metzen J H, Hutter F. Neural architecture search: a survey. The Journal of Machine Learning Research, 2019, 20(1): 1997–2017

    MathSciNet  Google Scholar 

  213. Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V. RoBERTa: a robustly optimized BERT pretraining approach. 2019, arXiv preprint arXiv: 1907.11692

    Google Scholar 

  214. Wang A, Singh A, Michael J, Hill F, Levy O, Bowman S R. GLUE: a multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. 2018, 353–355

    Chapter  Google Scholar 

  215. Renduchintala A, Konuk T, Kuchaiev O. Tied-LoRA: enhancing parameter efficiency of LoRA with weight tying. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 8694–8705

    Google Scholar 

  216. Hansen N, Ostermeier A. Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adaptation. In: Proceedings of the IEEE International Conference on Evolutionary Computation. 1996, 312–317

    Google Scholar 

  217. Ye M, Fang X, Du B, Yuen P C, Tao D. Heterogeneous federated learning: state-of-the-art and research challenges. ACM Computing Surveys, 2024, 56(3): 79

    Article  Google Scholar 

  218. Liu X Y, Zhu R, Zha D, Gao J, Zhong S, White M, Qiu M. Differentially private low-rank adaptation of large language model using federated learning. 2023, arXiv preprint arXiv: 2312.17493

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of Chian (Grant Nos. 62025206, 62302436, U23A20296), the Zhejiang Province’s “Lingyan” R&D Project (No. 2024C01259), and the Ningbo Science and Technology Special Projects (No. 2023Z212).

Author information

Authors and Affiliations

  1. School of Software Technology, Zhejiang University, Ningbo, 315000, China

    Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu & Yunjun Gao

  2. College of Computer Science and Technology, Zhejiang University, Hangzhou, 310007, China

    Yuren Mao & Yunjun Gao

Authors
  1. Yuren Mao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yuhang Ge
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yijiang Fan
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Wenyi Xu
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Yu Mi
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Zhonghao Hu
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Yunjun Gao
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Yunjun Gao.

Ethics declarations

Competing interests The authors declare that they have no competing interests or financial conflicts to disclose.

Additional information

Yuren Mao received his PhD degree under the supervision of Prof. Xuemin Lin in computer science from University of New South Wales, Australia in 2022. He is currently an assistant professor with the School of Software Technology, Zhejiang University, China. His current research interests include Large Language Models and its applications in Data Intelligence.

Yuhang Ge is currently working toward his PhD degree in the School of Software Technology at Zhejiang University, China. His research interests include Large Language Models and Data Management.

Yijiang Fan is currently studying as a master’s student in the School of Software Technology at Zhejiang University, China. His research interests include Large Language Models and collaborative inference.

Wenyi Xu is currently studying as a master’s student in the School of Software Technology at Zhejiang University, China. His research interests include Multimodal Large Models and RAG.

Yu Mi is currently studying as a master’s student in the School of Software Technology at Zhejiang University, China. Her research interests include Large Language Models and AI for science.

Zhonghao Hu is currently studying as a master’s student in the School of Software Technology at Zhejiang University, China. His research interests include Large Language Models and data discovery.

Yunjun Gao received the PhD degree in computer science from Zhejiang University, China, in 2008. He is currently a professor in the College of Computer Science and Technology, Zhejiang University, China. His research interests include Database, Big Data Management and Analytics, and AI interaction with DB technology.

Electronic supplementary material

A Survey on LoRA of Large Language Models

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mao, Y., Ge, Y., Fan, Y. et al. A survey on LoRA of large language models. Front. Comput. Sci. 19, 197605 (2025). https://doi.org/10.1007/s11704-024-40663-9

Download citation

  • Received: 02 July 2024

  • Accepted: 12 August 2024

  • Published: 14 December 2024

  • DOI: https://doi.org/10.1007/s11704-024-40663-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Keywords

  • low-rank adaptation
  • LoRA
  • large language models
  • LLMs
Use our pre-submission checklist

Avoid common mistakes on your manuscript.

Advertisement

Search

Navigation

  • Find a journal
  • Publish with us
  • Track your research

Discover content

  • Journals A-Z
  • Books A-Z

Publish with us

  • Journal finder
  • Publish your research
  • Open access publishing

Products and services

  • Our products
  • Librarians
  • Societies
  • Partners and advertisers

Our brands

  • Springer
  • Nature Portfolio
  • BMC
  • Palgrave Macmillan
  • Apress
  • Discover
  • Your US state privacy rights
  • Accessibility statement
  • Terms and conditions
  • Privacy policy
  • Help and support
  • Legal notice
  • Cancel contracts here

Not affiliated

Springer Nature

© 2025 Springer Nature