skip to main content
10.1145/3622896.3622906acmotherconferencesArticle/Chapter ViewAbstractPublication PagesccrisConference Proceedingsconference-collections
research-article

Instruction Tuning Text-to-SQL with Large Language Models in the Power Grid Domain

Published: 03 October 2023 Publication History

Abstract

This paper explores the large language models to address the Text-to-SQL task in real-world scenarios in the electricity domain. To tackle the lack of training data and corresponding databases for vertical domain real-world scenarios, the paper devised specific prompts to leverage ChatGPT for data generation, achieving significant improvements in annotation efficiency through automated data generation. Furthermore, to apply the powerful semantic parsing and generation capabilities of large language models to Text-to-SQL, the paper utilized a large language model for instruction tuning for SQL generation. This model has undergone secondary pre-training with electrical knowledge, tailoring it to the specific SQL generation task. On the power grid test set, the paper’s matching accuracy reached 65.7%, and the execution accuracy reached 80.9%. Additionally, the paper conducted further tests on various general large language models for zero-shot learning and single-sample prompt-based Text-to-SQL. The results indicate that while simple single-table queries can be achieved, meeting the requirements for complex queries remains challenging.

References

[1]
[1] Pengcheng Yin and Graham Neubig. 2018. TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Brussels, Belgium, 7–12.
[2]
[2] Parker Glenn, Parag Pravin Dakle, and Preethi Raghavan. 2023. Correcting Semantic Parses with Natural Language through Dynamic Schema Encoding. Retrieved June 27, 2023 from http://arxiv.org/abs/2305.19974
[3]
[3] Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation. Retrieved July 28, 2023 from http://arxiv.org/abs/1905.08205
[4]
[4] Kevin Lin, Ben Bogin, Mark Neumann, Jonathan Berant, and Matt Gardner. 2019. Grammar-based Neural Text-to-SQL Generation. Retrieved July 28, 2023 from http://arxiv.org/abs/1905.13326
[5]
[5] Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 9895–9901.
[6]
[6] Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, and Zhouhan Lin. 2022. RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 3215–3229. Retrieved July 28, 2023 from https://aclanthology.org/2022.emnlp-main.211
[7]
[7] Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 602–631. Retrieved July 28, 2023 from https://aclanthology.org/2022.emnlp-main.39
[8]
[8] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Retrieved July 28, 2023 from http://arxiv.org/abs/1910.10683
[9]
[9] Shuaichen Chang and Eric Fosler-Lussier. 2023. How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settings. Retrieved June 28, 2023 from http://arxiv.org/abs/2305.11853
[10]
[10] Brenden Lake and Marco Baroni. 2018. Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks. In Proceedings of the 35th International Conference on Machine Learning, PMLR, 2873–2882. Retrieved July 28, 2023 from https://proceedings.mlr.press/v80/lake18a.html
[11]
[11] Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. 2023. ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory. Retrieved June 30, 2023 from http://arxiv.org/abs/2306.03901
[12]
[12] Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. 2023. Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases. Retrieved July 28, 2023 from http://arxiv.org/abs/2303.14742
[13]
[13] Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020. Measuring Compositional Generalization: A Comprehensive Method on Realistic Data. Retrieved July 28, 2023 from http://arxiv.org/abs/1912.09713
[14]
[14] Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL. Retrieved July 28, 2023 from http://arxiv.org/abs/2302.05965
[15]
[15] Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R. Woodward, John Drake, and Qiaofu Zhang. 2021. Natural SQL: Making SQL Easier to Infer from Natural Language Specifications. Retrieved July 28, 2023 from http://arxiv.org/abs/2109.05153
[16]
[16] Ruoxi Sun, Sercan O. Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, and Tomas Pfister. 2023. SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL. Retrieved June 30, 2023 from http://arxiv.org/abs/2306.00739
[17]
[17] Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. Retrieved July 10, 2023 from http://arxiv.org/abs/2305.03111
[18]
[18] Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. 2023. A comprehensive evaluation of ChatGPT’s zero-shot Text-to-SQL capability. Retrieved June 28, 2023 from http://arxiv.org/abs/2303.13547
[19]
[19] Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. Retrieved June 28, 2023 from http://arxiv.org/abs/2304.11015
[20]
[20] Daking Rai, Bailin Wang, Yilun Zhou, and Ziyu Yao. 2023. Improving Generalization in Language Model-Based Text-to-SQL Semantic Parsing: Two Simple Semantic Boundary-Based Techniques. Retrieved June 27, 2023 from http://arxiv.org/abs/2305.17378
[21]
[21] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2019. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. Retrieved July 28, 2023 from http://arxiv.org/abs/1809.08887
[22]
[22] Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic Evaluation for Text-to-SQL with Distilled Test Suites.
[23]
[23] Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2021. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers. Retrieved July 28, 2023 from http://arxiv.org/abs/1911.04942
[24]
[24] Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca. Retrieved July 28, 2023 from http://arxiv.org/abs/2304.08177
[25]
[25] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. Retrieved July 28, 2023 from http://arxiv.org/abs/2103.10360
[26]
[26] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. 2022. GLM-130B: An Open Bilingual Pre-trained Model. Retrieved July 28, 2023 from http://arxiv.org/abs/2210.02414

Cited By

View all
  • (2025)Integration of LLM and Human–AI Coordination for Power Dispatching With Connected Electric Vehicles Under SAGVNsIEEE Transactions on Vehicular Technology10.1109/TVT.2024.343496974:2(1992-2002)Online publication date: Feb-2025
  • (2025)Deep generative models in energy system applications: Review, challenges, and future directionsApplied Energy10.1016/j.apenergy.2024.125059380(125059)Online publication date: Feb-2025
  • (2024)An Approach to the Analysis of Financial Documents Using Generative AIProceedings of the 7th International Conference on Networking, Intelligent Systems and Security10.1145/3659677.3659736(1-5)Online publication date: 18-Apr-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
CCRIS '23: Proceedings of the 2023 4th International Conference on Control, Robotics and Intelligent System
August 2023
215 pages
ISBN:9798400708190
DOI:10.1145/3622896
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 October 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. GPT
  2. Instruction Tuning
  3. Large Language Models
  4. Power Grid
  5. Text-To-SQL

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

CCRIS 2023

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)107
  • Downloads (Last 6 weeks)16
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Integration of LLM and Human–AI Coordination for Power Dispatching With Connected Electric Vehicles Under SAGVNsIEEE Transactions on Vehicular Technology10.1109/TVT.2024.343496974:2(1992-2002)Online publication date: Feb-2025
  • (2025)Deep generative models in energy system applications: Review, challenges, and future directionsApplied Energy10.1016/j.apenergy.2024.125059380(125059)Online publication date: Feb-2025
  • (2024)An Approach to the Analysis of Financial Documents Using Generative AIProceedings of the 7th International Conference on Networking, Intelligent Systems and Security10.1145/3659677.3659736(1-5)Online publication date: 18-Apr-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media