skip to main content
10.1145/3638530.3654104acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
poster

Local Search-based Approach for Cost-effective Job Assignment on Large Language Models

Published: 01 August 2024 Publication History

Abstract

Large Language Models (LLMs) have garnered significant attention due to their impressive capabilities. However, leveraging LLMs can be expensive due to the computational resources required, with costs depending on invocation numbers and input prompt lengths. Generally, larger LLMs deliver better performance but at a higher cost. In addition, prompts that provide more guidance to LLMs can increase the probability of correctly processing the job but also tend to be longer, increasing the processing cost. Therefore, selecting an appropriate LLM and prompt template is crucial for achieving an optimal trade-off between cost and performance. This paper formulates the job assignment on LLMs as a multi-objective optimisation problem and proposes a local search-based algorithm, termed LSAP, which aims to minimise the invocations cost while maximising overall performance. First, historical data is used to estimate the accuracy of each job submitted to a candidate LLM with a chosen prompt template. Subsequently, LSAP combines heuristic rules to select an appropriate LLM and prompt template based on the invocation cost and estimated accuracy. Extensive experiments on LLM-based log parsing, a typical software maintenance task that utilizes LLMs, demonstrate that LSAP can efficiently generate solutions with significantly lower cost and higher accuracy compared to the baselines.

References

[1]
2015. Welcome to PyGMO. Retrieved April 20, 2024 from https://esa.github.io/pygmo/index.html
[2]
Charles Audet, Jean Bigeon, Dominique Cartier, Sébastien Le Digabel, and Ludovic Salomon. 2021. Performance indicators in multiobjective optimization. European journal of operational research 292, 2 (2021), 397--422.
[3]
J. Blank and K. Deb. 2020. pymoo: Multi-Objective Optimization in Python. IEEE Access 8 (2020), 89497--89509.
[4]
Lingjiao Chen, Matei Zaharia, and James Zou. 2023. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv preprint arXiv:2305.05176 (2023).
[5]
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE transactions on evolutionary computation 6, 2 (2002), 182--197.
[6]
Yao Hao, Zhiqiu Huang, Hongjing Guo, and Guohua Shen. 2023. Test Input Selection for Deep Neural Network Enhancement Based on Multiple-Objective Optimization. In SANER. IEEE, 534--545.
[7]
Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R Lyu. 2021. A survey on automated log analysis for reliability engineering. ACM computing surveys (CSUR) 54, 6 (2021), 1--37.
[8]
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023. Large language models for software engineering: A systematic literature review. arXiv:2308.10620 (2023).
[9]
Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. 2024. Free-bloom: Zero-shot text-to-video generator with llm director and ldm animator. Advances in Neural Information Processing Systems 36 (2024).
[10]
Zhihan Jiang, Jinyang Liu, Zhuangbin Chen, Yichen Li, Junjie Huang, Yintong Huo, Pinjia He, Jiazhen Gu, and Michael R Lyu. 2023. LLMParser: A LLM-based Log Parsing Framework. arXiv:2310.01796 (2023).
[11]
Van-Hoang Le and Hongyu Zhang. 2023. Log Parsing: How Far Can ChatGPT Go?. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1699--1705.
[12]
Zeyang Ma, An Ran Chen, Dong Jae Kim, Tse-Hsun Peter Chen, and Shaowei Wang. 2024. LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing. In ICSE. 883--883.
[13]
Navid Sahebjamnia, Amir Mohammad Fathollahi-Fard, and Mostafa Hajiaghaei-Keshteli. 2018. Sustainable tire closed-loop supply chain network design: Hybrid metaheuristic algorithms for large-scale networks. Journal of cleaner production 196 (2018), 273--296.
[14]
Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, et al. 2023. All in one: Exploring unified video-language pre-training. In CVPR. 6598--6608.
[15]
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An Explanation of In-context Learning as Implicit Bayesian Inference. In ICLR.
[16]
Junjielong Xu, Ruichun Yang, Yintong Huo, Chengyu Zhang, and Pinjia He. 2023. Prompting for Automatic Log Template Extraction. arXiv:2307.09950 (2023).
[17]
Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, and Michael R Lyu. 2019. Tools and benchmarks for automated log parsing. In ICSE-SEIP. IEEE, 121--130.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
GECCO '24 Companion: Proceedings of the Genetic and Evolutionary Computation Conference Companion
July 2024
2187 pages
ISBN:9798400704956
DOI:10.1145/3638530
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 August 2024

Check for updates

Author Tags

  1. large language models
  2. job assignment
  3. local search
  4. log parsing

Qualifiers

  • Poster

Conference

GECCO '24 Companion
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 48
    Total Downloads
  • Downloads (Last 12 months)48
  • Downloads (Last 6 weeks)4
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media