Skip to main content

Advertisement

Cross-language few-shot intent recognition via prompt-based tuning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Cross-language intent recognition is a fundamental task in cross-language understanding. Recently, this task has been addressed by pretrained cross-language language models. Existing approaches typically augment pretrained language models with additional data, such as annotated parallel corpora. However, these additional data are scarce in practice, especially for low-resource languages. Inspired by the recent effective results of prompt learning, this paper proposes a new framework for enhancing cross-language few-shot intent recognition methods based on prompt tuning (CIRP). The proposed method converts the cross-language intent recognition task into a masked language modelling problem by designing prompt templates. To make the proposed model more generalizable, and avoid templates and label words dependent on a specific language, the method encodes the prompt templates into language-independent embedding representations via the multilingual pretrained language models, and initializes the label words into soft label words by averaging the [mask] vector values from different utterances of the same label, which reduces the distance between label word embeddings and encoder outputs of the [mask] to increase the accuracy of cross-language intent recognition. The experimental results on the few-shot cross-language MultiATIS++, MIvD benchmark dataset show that, compared with the four baseline models, the CIRP performs remarkably well in terms of intent recognition accuracy. Notably, when the sample sizes are set to 1 and 8 shots, the cross-language intent recognition accuracy metrics improve by an average of 11.75% compared with those of the baseline models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Algorithm 1
Algorithm 2
Fig. 3
Fig. 4

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of data and materials

The datasets and materials used during this study are available by following the links in the text.

Code availability

The Python code can be obtained by contacting the author (Yu Li).

References

  1. Bel N, Koster CHA, Villegas M (2003) Cross-lingual text categorization. In: Research and advanced technology for digital libraries, 7th European conference, ECDL 2003, Trondheim, Norway, August 17-22, 2003, Proceedings, vol 2769, pp 126–139

  2. Rigutini L, Maggini M, Liu B (2005) An EM based training algorithm for cross-language text categorization. In: 2005 IEEE WIC ACM International conference on web intelligence (WI 2005), 19–22 September 2005, Compiegne, France, pp 529–535

  3. Qin L, Ni M, Zhang Y, Che W (2020) Cosda-ml: Multi-lingual code-switching data augmentation for zero-shot cross-lingual NLP. In: Proceedings of the twenty-ninth international joint conference on artificial intelligence, IJCAI 2020, pp 3853–3860

  4. Devlin J, Chang M, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2–7, 2019, vol 1 (Long and Short Papers), pp 4171–4186

  5. Liu X, Zheng Y, Du Z, Din M, Qian Y, Yang Z, Tang J (2023) Gpt understands, too. AI Open

  6. Liu P, Yuan W, Fu J, Jiang Z, Hayashi H, Neubig G (2023) Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput Surv 55(9):195–119535

    Article  Google Scholar 

  7. Li D, Zhang T, Deng J, Huang L, Wang C, He X, Xue H (2024) Unipsda: Unsupervised pseudo semantic data augmentation for zero-shot cross-lingual natural language understanding, pp 17062–17073

  8. Jiang G, Liu S, Zhao Y, Sun Y, Zhang M (2022) Fake news detection via knowledgeable prompt learning. 59:103029

    Google Scholar 

  9. Davison J, Feldman J, Rush AM (2019) Commonsense knowledge mining from pretrained models. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3–7, 2019, pp 1173–1178

  10. Han X, Zhao W, Ding N, Liu Z, Sun M (2022) PTR: prompt tuning with rules for text classification. AI Open. 3:182–192

    Article  MATH  Google Scholar 

  11. He K, Huang Y, Mao R, Gong T, Li C, Cambria E (2023) Virtual prompt pre-training for prototype-based few-shot relation extraction. Expert Syst Appl 213:118927

    Article  Google Scholar 

  12. Liu J, Wu K, Nie Q, Chen Y, Gao B, Liu Y, Wang J, Wang C, Zheng F (2024) Unsupervised continual anomaly detection with contrastively-learned prompt. In: Thirty-eighth AAAI conference on artificial intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp 3639–3647

  13. Ding N, Chen Y, Han X, Xu G, Wang X, Xie P, Zheng H, Liu Z, Li J, Kim H (2022) Prompt-learning for fine-grained entity typing. In: Findings of the association for computational linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7–11, 2022, pp 6888–6901

  14. Xu Y, Zhang L, Zhou D (2024) TECA: A two-stage approach with controllable attention soft prompt for few-shot nested named entity recognition. In: Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation, LREC/COLING 2024, 20–25 May, 2024, Torino, Italy, pp 15698–15710

  15. Hu S, Ding N, Wang H, Liu Z, Wang J, Li J, Wu W, Sun M (2022) Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. In: Proceedings of the 60th annual meeting of the association for computational linguistics (vol 1: Long Papers), ACL 2022, Dublin, Ireland, May 22–27, 2022, pp 2225–2240

  16. Shin T, Razeghi Y, IV RLL, Wallace E, Singh S (2020) Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In: Proceedings of the 2020 Conference on empirical methods in natural language processing, EMNLP 2020, Online, November 16–20, 2020, pp 4222–4235

  17. Hambardzumyan K, Khachatrian H, May J (2021) WARP: word-level adversarial reprogramming. In: Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing, ACL/IJCNLP 2021, (vol 1: Long Papers), Virtual Event, August 1–6, 2021, pp 4921–4933

  18. Conneau A, Lample G (2019) Cross-lingual language model pretraining. In: Advances in neural information processing systems 32: annual conference on neural information processing systems 2019, NeurIPS 2019, December 8–14, 2019, Vancouver, BC, Canada, pp 7057–7067

  19. Qin L, Chen Q, Xie T, Li Q, Lou J, Che W, Kan M (2022) Gl-clef: A global-local contrastive learning framework for cross-lingual spoken language understanding. In: Proceedings of the 60th annual meeting of the association for computational linguistics (vol 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp 2677–2686

  20. Min S, Lewis M, Hajishirzi H, Zettlemoyer L (2022) Noisy channel language model prompting for few-shot text classification. In: Proceedings of the 60th annual meeting of the association for computational linguistics (vol 1: Long Papers), ACL 2022, Dublin, Ireland, May 22–27, 2022, pp 5316–5330

  21. Li X, Fang L, Zhang L, Cao P (2023) An interactive framework of cross-lingual NLU for in-vehicle dialogue. Sensors. 23(20):8501

    Article  MATH  Google Scholar 

  22. Rafiepour M, Sartakhti JS (2023) CTRAN: cnn-transformer-based network for natural language understanding. Eng Appl Artif Intell 126:107013

    Article  MATH  Google Scholar 

  23. Ruder S, Vulic I, Søgaard A (2019) A survey of cross-lingual word embedding models. J. Artif. Intell. Res. 65:569–631

    Article  MathSciNet  MATH  Google Scholar 

  24. Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359

    Article  MATH  Google Scholar 

  25. Mairesse F, Gasic M, Jurcícek F, Keizer S, Thomson B, Yu K, Young SJ (2009) Spoken language understanding from unaligned data using discriminative classification models. In: Proceedings of the IEEE international conference on acoustics, speech, and signal processing, ICASSP 2009, 19–24 April 2009, Taipei, Taiwan, pp 4749–4752

  26. Adams O, Makarucha AJ, Neubig G, Bird S, Cohn T (2017) Cross-lingual word embeddings for low-resource language modeling. In: Lapata M, Blunsom P, Koller A (eds) Proceedings of the 15th conference of the European chapter of the association for computational linguistics, EACL 2017, Valencia, Spain, April 3–7, 2017, vol 1: Long Papers, pp 937–947

  27. Bayer M, Kaufhold M, Reuter C (2023) A survey on data augmentation for text classification. ACM Comput Surv 55(7):146–114639

    Article  MATH  Google Scholar 

  28. Kim H, Komachi M (2023) Enhancing few-shot cross-lingual transfer with target language peculiar examples. In: Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9–14, 2023, pp 747–767

  29. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21:140–114067

    MathSciNet  Google Scholar 

  30. Xia C, Zhang C, Nguyen H, Zhang J, Yu P (2020) Cg-bert: Conditional text generation with bert for generalized few-shot intent detection. arXiv preprint arXiv:2004.01881

  31. Guo X, Adnan HM, Abidin MZZ (2024) Detecting offensive language on malay social media: A zero-shot, cross-language transfer approach using dual-branch mbert. Appl Sci 14(13):5777

    Article  Google Scholar 

  32. Pouramini A, Faili H (2024) Matching tasks to objectives: Fine-tuning and prompt-tuning strategies for encoder-decoder pre-trained language models. Appl Intell pp 9783–9810

  33. Qi K, Wan H, Du J, Chen H (2022) Enhancing cross-lingual natural language inference by prompt-learning from cross-lingual templates. In: Proceedings of the 60th annual meeting of the association for computational linguistics (vol 1: Long Papers), ACL 2022, Dublin, Ireland, May 22–27, 2022, pp 1910–1923

  34. Zheng C, Huang M (2021) Exploring prompt-based few-shot learning for grounded dialog generation. arXiv preprint arXiv:2109.06513

  35. Du Y, Yin Z, Xie R, Zhang Q (2024) Prompt template construction by average gradient search with external knowledge for aspect sentimental analysis. Expert Syst Appl 238(Part F), p 122271

  36. Schick T, Schütze H (2021) Exploiting cloze-questions for few-shot text classification and natural language inference. In: Proceedings of the 16th conference of the European chapter of the association for computational linguistics: main volume, EACL 2021, Online, April 19–23, 2021, pp 255–269

  37. Li XL, Liang P (2021) Prefix-tuning: Optimizing continuous prompts for generation. In: Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing, ACL/IJCNLP 2021, (vol 1: Long Papers), Virtual Event, August 1–6, 2021, pp 4582–4597

  38. Liu X, Ji K, Fu Y, Tam WL, Du Z, Yang Z, Tang J (2021) P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602

  39. Zhao M, Schütze H (2021) Discrete and soft prompting for multilingual models. In: Proceedings of the 2021 conference on empirical methods in natural language processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7–11 November, 2021, pp 8547–8555

  40. Lin XV, Mihaylov T, Artetxe M, Wang T, Chen S, Simig D, Ott M, Goyal N, Bhosale S, Du J et al (2021) Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.10668

  41. Huang L, Ma S, Zhang D, Wei F, Wang H (2022) Zero-shot cross-lingual transfer of prompt-based tuning with a unified multilingual prompt. In: Proceedings of the 2022 conference on empirical methods in natural language processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7–11, 2022, pp 11488–11497

  42. Price P (1990) Evaluation of spoken language systems: The atis domain. In: Speech and natural language: proceedings of a workshop held at hidden valley, Pennsylvania, June 24–27, 1990

  43. Upadhyay S, Faruqui M, Tür, G, Hakkani-Tür D, Heck LP (2018) (almost) zero-shot cross-lingual spoken language understanding. In: 2018 IEEE International conference on acoustics, speech and signal processing, ICASSP 2018, Calgary, AB, Canada, April 15–20, 2018, pp 6034–6038

  44. Xu W, Haider B, Mansour S (2020) End-to-end slot alignment and recognition for cross-lingual nlu. arXiv preprint arXiv:2004.14353

  45. Zheng J, Fan F, Li J (2024) Incorporating lexical and syntactic knowledge for unsupervised cross-lingual transfer. In: Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation, LREC/COLING 2024, 20–25 May, 2024, Torino, Italy, pp 8986–8997

Download references

Author information

Authors and Affiliations

Authors

Contributions

Xinlu Li: Conceptualization, Methodology, Supervision. Yu Li : Software, Conducting experiments, Writing - Original draft preparation. Pei Cao: Reviewing, Investigation and Editing.

Corresponding author

Correspondence to Xinlu Li.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A Section title of first appendix

Appendix A Section title of first appendix

An appendix contains supplementary information that is not an essential part of the text itself but which may be helpful in providing a more comprehensive understanding of the research problem or it is information that is too cumbersome to be included in the body of the paper.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cao, P., Li, Y. & Li, X. Cross-language few-shot intent recognition via prompt-based tuning. Appl Intell 55, 60 (2025). https://doi.org/10.1007/s10489-024-06089-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-06089-3

Keywords