Skip to main content

S3LLM: Large-Scale Scientific Software Understanding with LLMs Using Source, Metadata, and Document

  • Conference paper
  • First Online:
Computational Science – ICCS 2024 (ICCS 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14834))

Included in the following conference series:

  • 625 Accesses

Abstract

The understanding of large-scale scientific software is a significant challenge due to its diverse codebase, extensive code length, and target computing architectures. The emergence of generative AI, specifically large language models (LLMs), provides novel pathways for understanding such complex scientific codes. This paper presents S3LLM, an LLM-based framework designed to enable the examination of source code, code metadata, and summarized information in conjunction with textual technical reports in an interactive, conversational manner through a user-friendly interface. S3LLM leverages open-source LLaMA-2 models to enhance code analysis through the automatic transformation of natural language queries into domain-specific language (DSL) queries. In addition, S3LLM is equipped to handle diverse metadata types, including DOT, SQL, and customized formats. Furthermore, S3LLM incorporates retrieval-augmented generation (RAG) and LangChain technologies to directly query extensive documents. S3LLM demonstrates the potential of using locally deployed open-source LLMs for the rapid understanding of large-scale scientific computing software, eliminating the need for extensive coding expertise and thereby making the process more efficient and effective. S3LLM is available at https://github.com/ResponsibleAILab/s3llm.

This manuscript has been authored by UT-Battelle, LLC under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://en.wikipedia.org/wiki/DOT_(graph_description_language).

  2. 2.

    A software toolkit for porting E3SM land models onto GPUs using OpenACC.

References

  1. Doxygen. https://www.doxygen.nl/. Accessed 01 Mar 2024

  2. Acharya, A., Singh, B., Onoe, N.: LLM based generation of item-description for recommendation system. In: Proceedings of the 17th ACM RecSys, pp. 1204–1207 (2023)

    Google Scholar 

  3. Bill, D., Eriksson, T.: Fine-tuning a LLM using reinforcement learning from human feedback for a therapy chatbot application (2023)

    Google Scholar 

  4. Chang, Y., Lo, K., Goyal, T., Iyyer, M.: Booookscore: a systematic exploration of book-length summarization in the era of LLMs. arXiv preprint arXiv:2310.00785 (2023)

  5. Freire, S.K., Wang, C., Niforatos, E.: Chatbots in knowledge-intensive contexts: comparing intent and LLM-based systems. arXiv preprint arXiv:2402.04955 (2024)

  6. Golaz, J.-C., et al.: The DOE E3SM model version 2: overview of the physical model and initial model evaluation. J. Adv. Model. Earth Syst. 14(12), e2022MS003156 (2022)

    Google Scholar 

  7. Huang, S., Zhao, J., Li, Y., Wang, L.: Learning preference model for LLMs via automatic preference data generation. In: The 2023 Conference on EMLP (2023)

    Google Scholar 

  8. Jury, B., Lorusso, A., Leinonen, J., Denny, P., Luxton-Reilly, A.: Evaluating LLM-generated worked examples in an introductory programming course. In: Proceedings of the 26th Australasian Computing Education Conference, pp. 77–86 (2024)

    Google Scholar 

  9. Lewis, P., et al.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474 (2020)

    Google Scholar 

  10. Oleson, K.W., Lawrence, D.M., et al.: Technical description of version 4.0 of the community land model (CLM). NCAR Tech. Note NCAR/TN-478+ STR 257, 1–257 (2010)

    Google Scholar 

  11. Schwartz, P., Wang, D., Yuan, F., Thornton, P.: SPEL: software tool for porting E3SM land model with openacc in a function unit test framework. In: 2022 Workshop on Accelerator Programming Using Directives (WACCPD), pp. 43–51. IEEE (2022)

    Google Scholar 

  12. Tsigkanos, C., Rani, P., Müller, S., Kehrer, T.: Variable discovery with large language models for metamorphic testing of scientific software. In: Mikyška, J., de Mulatier, C., Paszynski, M., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds.) ICCS 2023. LNCS, vol. 14073, pp. 321–335. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-35995-8_23

    Chapter  Google Scholar 

  13. Van Veen, D., Van Uden, C., et al.: Clinical text summarization: adapting large language models can outperform human experts. Res. Square (2023)

    Google Scholar 

  14. Zhang, T., Ladhak, F., et al.: Benchmarking large language models for news summarization. Trans. Assoc. Comput. Linguist. 12, 39–57 (2024)

    Article  Google Scholar 

  15. Zheng, L., Chiang, W.-L., et al.: Judging LLM-as-a-judge with MT-bench and chatbot arena. In: Advances in Neural Information Processing Systems, vol. 36 (2024)

    Google Scholar 

  16. Zheng, W., Wang, D., Song, F.: XScan: an integrated tool for understanding open source community-based scientific code. In: Rodrigues, J.M.F., et al. (eds.) ICCS 2019. LNCS, vol. 11536, pp. 226–237. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22734-0_17

    Chapter  Google Scholar 

  17. Zheng, W., Wang, D., Song, F.: FQL: an extensible feature query language and toolkit on searching software characteristics for HPC applications. In: Tools and Techniques for High Performance Computing, pp. 129–142 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yunhe Feng .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shaik, K. et al. (2024). S3LLM: Large-Scale Scientific Software Understanding with LLMs Using Source, Metadata, and Document. In: Franco, L., de Mulatier, C., Paszynski, M., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M.A. (eds) Computational Science – ICCS 2024. ICCS 2024. Lecture Notes in Computer Science, vol 14834. Springer, Cham. https://doi.org/10.1007/978-3-031-63759-9_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-63759-9_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-63758-2

  • Online ISBN: 978-3-031-63759-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics