Text2VQL: Teaching a Model Query Language to Open-Source Language Models with ChatGPT
Pages 13 - 24
Abstract
While large language models (LLMs) like ChatGPT has demonstrated impressive capabilities in addressing various software engineering tasks, their use in a model-driven engineering (MDE) context is still in an early stage. Since the technology is proprietary and accessible solely through an API, its use may be incompatible with the strict protection of intellectual properties in industrial models. While there are open-source LLM alternatives, they often lack the power of proprietary models and require extensive data fine-tuning to realize their full potential. Furthermore, open-source datasets tailored for MDE tasks are scarce, posing challenges for training such models effectively.
In this work, we introduce Text2VQL, a framework that generates graph queries captured in the VIATRA Query Language (VQL) from natural language specifications using open-source LLMs. Initially, we create a high-quality synthetic dataset comprising pairs of queries and their corresponding natural language descriptions using ChatGPT and VIATRA parser. Leveraging this dataset, we use parameter-efficient tuning to specialize three open-source LLMs, namely, DeepSeek Coder 1b, DeepSeek Coder 7b, and CodeLlama 7b for VQL query generation. Our experimental evaluation demonstrates that the fine-tuned models outperform the base models in query generation, highlighting the usefulness of our synthetic dataset. Moreover, one of the fine-tuned models achieves performance comparable to ChatGPT.
References
[1]
Seif Abukhalaf, Mohammad Hamdaqa, and Foutse Khomh. 2023. On codex prompt engineering for OCL generation: an empirical study. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). IEEE, 148--157.
[2]
Gábor Bergmann, István Dávid, Ábel Hegedüs, Ákos Horváth, István Ráth, Zoltán Ujhelyi, and Dániel Varró. 2015. Viatra 3: A Reactive Model Transformation Platform. In ICMT (LNCS, Vol. 9152). Springer, 101--110.
[3]
Gábor Bergmann, Ákos Horváth, István Ráth, Dániel Varró, András Balogh, Zoltán Balogh, and András Ökrös. 2010. Incremental Evaluation of Model Queries over EMF Models. In MoDELS (1) (LNCS, Vol. 6394). Springer, 76--90.
[4]
Gábor Bergmann, András Ökrös, István Ráth, Dániel Varró, and Gergely Varró. 2008. Incremental pattern matching in the viatra model transformation system. In GRaMoT@ICSE. ACM, 25--32.
[5]
Gábor Bergmann, Zoltán Ujhelyi, István Ráth, and Dániel Varró. 2011. A Graph Query Language for EMF Models. In ICMT@TOOLS (LNCS, Vol. 6707). Springer, 167--182.
[6]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. In NeurIPS.
[7]
Jordi Cabot, David Delgado, and Lola Burgueño. 2022. Combining OCL and natural language: a call for a community effort. In MoDELS (Companion). ACM, 908--912.
[8]
Javier Cámara, Javier Troya, Lola Burgueño, and Antonio Vallecillo. 2023. On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML. Softw. Syst. Model. 22, 3 (2023), 781--793.
[9]
Meriem Ben Chaaben, Lola Burgueño, and Houari A. Sahraoui. 2023. Towards using Few-Shot Prompt Learning for Automating Model Completion. In ICSE (NIER). IEEE, 7--12.
[10]
Sahil Chaudhary. 2023. Code Alpaca: An Instruction-following LLaMA model for code generation. https://github.com/sahil280114/codealpaca.
[11]
Boqi Chen, Kua Chen, Shabnam Hassani, Yujing Yang, Daniel Amyot, Lysanne Lessard, Gunter Mussbacher, Mehrdad Sabetzadeh, and Dániel Varró. 2023. On the Use of GPT-4 for Creating Goal Models: An Exploratory Study. In REW. IEEE, 262--271.
[12]
Hailin Chen, Fangkai Jiao, Xingxuan Li, Chengwei Qin, Mathieu Ravaut, Ruochen Zhao, Caiming Xiong, and Shafiq Joty. 2023. ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up? CoRR abs/2311.16989 (2023).
[13]
Kua Chen, Yujing Yang, Boqi Chen, José Antonio Hernández López, Gunter Mussbacher, and Dániel Varró. 2023. Automated Domain Modeling with Large Language Models: A Comparative Study. In MoDELS. IEEE, 162--172.
[14]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. CoRR abs/2107.03374 (2021).
[15]
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234 (2022).
[16]
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming-The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024).
[17]
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR. OpenReview.net.
[18]
John Hutchinson, Jon Whittle, Mark Rouncefield, and Steinar Kristoffersen. 2011. Empirical assessment of MDE in industry. In Proceedings of the 33rd international conference on software engineering. 471--480.
[19]
George Katsogiannis-Meimarakis and Georgia Koutrika. 2021. A deep dive into deep learning approaches for text-to-sql systems. In Proceedings of the 2021 International Conference on Management of Data. 2846--2851.
[20]
Dimitris Kolovos. [n. d.]. Metamodelling with ChatGPT. https://www-users.york.ac.uk/dimitris.kolovos/blog/metamodelling-with-chatgpt/.
[21]
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems 32 (2019).
[22]
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. StarCoder: may the source be with you! CoRR abs/2305.06161 (2023).
[23]
Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In ACL/IJCNLP (1). Association for Computational Linguistics, 4582--4597.
[24]
Tiedong Liu and Bryan Kian Hsiang Low. 2023. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks. CoRR abs/2305.14201 (2023).
[25]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11692 (2019).
[26]
José Antonio Hernández López and Jesús Sánchez Cuadrado. 2020. MAR: a structure-based search engine for models. In Proceedings of the 23rd ACM/IEEE international conference on model driven engineering languages and systems. 57--67.
[27]
José Antonio Hernández López and Jesús Sánchez Cuadrado. 2022. An efficient and scalable search engine for models. Software and Systems Modeling 21, 5 (2022), 1715--1737.
[28]
José Antonio Hernández López, Javier Luis Cánovas Izquierdo, and Jesús Sánchez Cuadrado. 2022. ModelSet: a dataset for machine learning in model-driven engineering. Softw. Syst. Model. 21, 3 (2022), 967--986.
[29]
José Antonio Hernández López, Riccardo Rubei, Jesús Sánchez Cuadrado, and Davide Di Ruscio. 2022. Machine learning methods for model classification: a comparative study. In MoDELS. ACM, 165--175.
[30]
Kristóf Marussy, Attila Ficsor, Oszkár Semeráth, and Dániel Varró. 2024. Refinery: Graph Solver as a Service. In ICSE: Tool Demonstration Track. In press.
[31]
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2024. Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey. ACM Comput. Surv. 56, 2 (2024), 30:1--30:40.
[32]
Gunter Mussbacher, Benoit Combemale, J'org Kienzle, Silvia Abrah ao, Hyacinth Ali, Nelly Bencomo, Márton Búr, Loli Burgue no, Gregor Engels, Pierre Jeanjean, Jean-Marc Jézéquel, Thomas Kühn, Sébastien Mosser, Houari Sahraoui, Eugene Syriani, Dániel Varró, and Martin Weyssow. 2020. Opportunities in Intelligent Modeling Assistance. Softw. Syst. Model. 19, 5 (2020), 1045--1053.
[33]
OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023).
[34]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In NeurIPS.
[35]
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code Llama: Open Foundation Models for Code. CoRR abs/2308.12950 (2023).
[36]
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. CoRR abs/2211.05100 (2022).
[37]
Oszkár Semeráth, Aren A. Babikian, Sebastian Pilarski, and Dániel Varró. 2019. Viatra solver: a framework for the automated generation of consistent domain-specific models. In ICSE (Companion Volume). IEEE / ACM, 43--46.
[38]
Dave Steinberg, Frank Budinsky, Ed Merks, and Marcelo Paternostro. 2008. EMF: eclipse modeling framework. Pearson Education.
[39]
Gábor Szárnyas, Benedek Izsó, István Ráth, and Dániel Varró. 2018. The Train Benchmark: cross-technology performance evaluation of continuous model queries. Softw. Syst. Model. 17, 4 (2018), 1365--1393.
[40]
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca.
[41]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. CoRR abs/2302.13971 (2023).
[42]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. CoRR abs/2307.09288 (2023).
[43]
Zoltán Ujhelyi, Gábor Bergmann, Ábel Hegedüs, Ákos Horváth, Benedek Izsó, István Ráth, Zoltán Szatmári, and Dániel Varró. 2015. EMF-IncQuery: An integrated development environment for live model queries. Sci. Comput. Program. 98 (2015), 80--99.
[44]
Dániel Varró, Gábor Bergmann, Ábel Hegedüs, Ákos Horváth, István Ráth, and Zoltán Ujhelyi. 2016. Road to a reactive and incremental model transformation platform: three generations of the VIATRA framework. Softw. Syst. Model. 15, 3 (2016), 609--629.
[45]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In NIPS. 5998--6008.
[46]
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In ACL (1). Association for Computational Linguistics, 13484--13508.
[47]
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent Abilities of Large Language Models. Trans. Mach. Learn. Res. 2022 (2022).
[48]
Martin Weyssow, Houari A. Sahraoui, and Eugene Syriani. 2022. Recommending metamodel concepts during modeling activities with pre-trained language models. Softw. Syst. Model. 21, 3 (2022), 1071--1089.
[49]
Jon Whittle, John Hutchinson, and Mark Rouncefield. 2013. The state of practice in model-driven engineering. IEEE software 31, 3 (2013), 79--85.
[50]
Canwen Xu, Daya Guo, Nan Duan, and Julian J. McAuley. 2023. Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data. In EMNLP. Association for Computational Linguistics, 6268--6278.
[51]
Song Yang and Houari A. Sahraoui. 2022. Towards automatically extracting UML class diagrams from natural language specifications. In MoDELS (Companion). ACM, 396--403.
[52]
Önder Babur. 2019. A labeled Ecore metamodel dataset for domain clustering. https://doi.org/10.5281/zenodo.2585432
Recommendations
ModelMate: A recommender for textual modeling languages based on pre-trained language models
MODELS '24: Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and SystemsCurrent DSL environments lack smart editing facilities intended to enhance modeler productivity and cannot keep pace of current developments of integrated development environments based on AI. In this paper, we propose an approach to address this ...
AI-Driven Consistency of SysML Diagrams
MODELS '24: Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and SystemsGraphical modeling languages, expected to simplify systems analysis and design, present a challenge in maintaining consistency across their varied views. Traditional rule-based methods for ensuring consistency in languages like UML often fall short in ...
Automated Derivation of UML Sequence Diagrams from User Stories: Unleashing the Power of Generative AI vs. a Rule-Based Approach
MODELS '24: Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and SystemsUser stories are informal, non-technical descriptions of features from a user's perspective that guide collaboration and iterative development in Agile projects. However, ambiguities in user stories can lead to miscommunication among stakeholders. Design ...
Comments
Information & Contributors
Information
Published In
September 2024
311 pages
ISBN:9798400705045
DOI:10.1145/3640310
- General Chairs:
- Alexander Egyed,
- Manuel Wimmer,
- Program Chairs:
- Marsha Chechik,
- Benoit Combemale
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.
Sponsors
- Johannes Kepler University, Linz, Austria
- SIGSOFT: ACM Special Interest Group on Software Engineering
- IEEE CS
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 22 September 2024
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
MODELS '24
Sponsor:
MODELS '24: ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems
September 22 - 27, 2024
Linz, Austria
Acceptance Rates
MODELS '24 Paper Acceptance Rate 26 of 124 submissions, 21%;
Overall Acceptance Rate 144 of 506 submissions, 28%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 488Total Downloads
- Downloads (Last 12 months)488
- Downloads (Last 6 weeks)172
Reflects downloads up to 03 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in