Skip to main content
SpringerLink
Log in

Evaluating ChatGPT’s Proficiency in Understanding and Answering Microservice Architecture Queries Using Source Code Insights

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Microservice architecture has become increasingly prevalent in the software domain due to its inherent flexibility, scalability, and enhanced deployment capabilities. However, the shift towards this architecture has also introduced complexities in understanding software systems, particularly when grappling with Enterprise Architecture documentation developed by different teams. While there have been efforts to make comprehension easier through automated code summaries and visual displays, developers still need to make substantial efforts to grasp the system entirely. This study investigates the potential of Large Language Models, such as ChatGPT, to answer intricate queries related to cloud-native systems, leveraging both source code and a system intermediate representation obtained from static code analysis. We focused on assessing ChatGPT’s proficiency in answering questions focused on the service and interaction perspectives of microservice systems. Our results highlight the advantages of integrating intermediate representations of the system for a richer context while addressing the model’s inherent limitations. This work sets the stage for future studies to explore how Large Language Models can make understanding software easier in dynamic, microservice-heavy settings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availability Statement

We have shared our open-source tools, datasets and intermediate representations from involved tools through a third-party Zenodo package. Our code with full implementation of every step in the pipeline is available at: https://github.com/cloudhubs/Microservice-AI-Reasoning. The data extraction from the train ticket project and the designed questions csvs are available at: https://zenodo.org/record/8358519.

Notes

  1. TrainTicket v0.1.0: https://github.com/FudanSELab/train-ticket/tree/0.1.0.

  2. Spring Boot: https://spring.io/projects/spring-boot.

  3. Cloc: https://github.com/AlDanial/cloc.

  4. Tiktoken: https://github.com/openai/tiktoken.

  5. Prototype: https://github.com/cloudhubs/Microservice-AI-Reasoning.

  6. Dataset: https://zenodo.org/record/8358519.

  7. Apache Commons IO: https://commons.apache.org/proper/commons-io/index.html.

  8. Javaparser: https://javaparser.org/.

  9. https://github.com/cloudhubs/Distributed-Systems-Semantic-Clone-Detector.

  10. https://github.com/cloudhubs/authz-flow-analysis.

  11. https://github.com/cloudhubs/Microservice-AI-Reasoning.

  12. Jupyter: https://jupyter.org/.

  13. Pandas Lib:https://pandas.pydata.org/.

  14. https://zenodo.org/record/8358519.

  15. OpenAI: https://github.com/openai/openai-python.

  16. https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4.

  17. https://github.com/cloudhubs/Microservice-AI-Reasoning.

References

  1. Parker G, et al. Visualizing anti-patterns in microservices at runtime: a systematic mapping study. IEEE Access. 2023.

  2. Abdelfattah AS. Microservices-based systems visualization: student research abstract. In: Proceedings of the 37th ACM/SIGAPP symposium on applied computing (SAC ’22). New York: Association for Computing Machinery; 2022. p. 1460–1464. https://doi.org/10.1145/3477314.3506963.

  3. Taibi D, Lenarduzzi V, Pahl C, Janes A. Microservices in agile software development: a workshop-based study into issues, advantages, and disadvantages. In: Proceedings of the XP2017 scientific workshops (XP ’17). New York: Association for Computing Machinery; 2017. p. 1–5. https://doi.org/10.1145/3120459.3120483.

  4. Abdelfattah AS, Cerny T. Roadmap to reasoning in microservice systems: a rapid review. Appl Sci. 2023;13:1838. https://doi.org/10.3390/app13031838.

  5. Mayer B, Weinreich R. An approach to extract the architecture of microservice-based software systems. In: 2018 IEEE symposium on service-oriented system engineering (SOSE). IEEE; 2018, p. 21–30.

  6. Toffetti G, Brunner S, Blöchlinger M, Dudouet F, Edmonds A. An architecture for self-managing microservices. In: Proceedings of the 1st international workshop on automated incident management in cloud. 2015. p. 19–24.

  7. Calçado P. Building products at soundcloudpart III: microservices in scala and finagle. Technical report, SoundCloud Limited; 2014.

  8. Kleehaus M, Matthes F. Challenges in documenting microservice-based IT landscape: a survey from an enterprise architecture management perspective. In: Proc. - 2019 IEEE 23rd Int. Enterp. Distrib. Object Comput. Conf. EDOC 2019. 2019. p. 11–20.

  9. Bogner J, Fritzsch J, Wagner S, Zimmermann A. Industry practices and challenges for the evolvability assurance of microservices. Empir Soft Eng. 2021;26:104.

    Article  Google Scholar 

  10. Cerny T, Taibi D. Microservice-aware static analysis: opportunities, 1 gaps, and advancements. In: Joint post-proceedings of the third and forth international conference on microservices, microservices 2020/2022; 2023.

  11. Zhang C, et al. A survey of automatic source code summarization. Symmetry. 2022;14. https://www.mdpi.com/2073-8994/14/3/471.

  12. Geng M, Wang S, Dong D, Wang H, Cao S, Zhang K, Jin Z. Interpretation-based code summarization. In: Proceedings of the 2023 IEEE/ACM 31st international conference on program comprehension (ICPC), Melbourne, VI, Australia, 15–16 May 2023.

  13. Zhao WX, Zhou K, Li J, Tang T, Wang X, Hou Y, Min Y, Zhang B, Zhang J, Dong Z, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023).

  14. Hong S, Zheng X, Chen J, Cheng Y, Wang J, Zhang C, Wang Z, Yau S K S, Lin Z, Zhou L, et al. Metagpt: meta programming for multi-agent collaborative framework. 2023. arXiv preprint arXiv:2308.00352.

  15. Kaddour J, Harris J, Mozes M, Bradley H, Raileanu R, McHardy R. Challenges and applications of large language models. 2023. arXiv preprint arXiv:2307.10169.

  16. Dong Q, Li L, Dai D, Zheng C, Wu Z, Chang B, Sun X, Xu J, Sui Z. A survey on in-context learning. 2022. arXiv preprint arXiv:2301.00234.

  17. Walker A, Laird I, Cerny T. On automatic software architecture reconstruction of microservice applications. Singapore: Springer Singapore; 2021. p. 223–34.

    Google Scholar 

  18. Bushong V, Das D, Cerny T. Reconstructing the holistic architecture of microservice systems using static analysis. In: Proceedings of the 12th international conference on cloud computing and services science—CLOSER. 2022. p. 149–57. https://doi.org/10.5220/0011032100003200.

  19. Abdelfattah A, Schiewe M, Curtis J, Cerny T, Song E. Towards security-aware microservices: on extracting endpoint data access operations to determine access rights. In: 13th international conference on cloud computing and services science (CLOSER 2023). 2023.

  20. Zhou X, et al. Poster: benchmarking microservice systems for software engineering research. In: 2018 IEEE/ACM 40th international conference on software engineering: companion (ICSE-companion); 2018.

  21. Taibi D, Lenarduzzi V, Ahmad MO, Liukkunen K. Comparing communication effort within the scrum, scrum with kanban, xp, and banana development processes. In: Proceedings of the 21st international conference on evaluation and assessment in software engineering. 2017. p. 258–63.

  22. Cerny T, Abdelfattah AS, Bushong V, Al Maruf A, Taibi D. Microservice architecture reconstruction and visualization techniques: a review. In: 2022 IEEE international conference on service-oriented system engineering (SOSE). IEEE; 2022. p. 39–48.

  23. Morais G, Bork D, Adda M. Towards an ontology-driven approach to model and analyze microservices architectures. In: Proceedings of the 13th international conference on management of digital ecosystems. 2021. p. 79–86.

  24. Haselböck S, Weinreich R, Buchgeher G. Using decision models for documenting microservice architectures: a student experiment and focus group study. In: 2019 IEEE international conference on service-oriented system engineering (SOSE). IEEE; 2019. p. 37–3709.

  25. Barabanov AV. A method for collecting security-specific architectural information for microservice-based systems for design security assessment. 2021.

  26. Wang S, Tian Y, He D. gDoc: automatic generation of structured API documentation. In: Companion proceedings of the ACM web conference 2023. 2023. p. 53–6.

  27. Granchelli G, Cardarelli M, Di Francesco P, Malavolta I, Iovino L, Di Salle A. Microart: a software architecture recovery tool for maintaining microservice-based systems. In: 2017 IEEE international conference on software architecture workshops (ICSAW). IEEE; 2017. p. 298–302.

  28. Kleehaus M, Uludağ Ö, Schäfer P, Matthes F. MICROLYZE: a framework for recovering the software architecture in microservice-based environments. Berlin: Springer; 2018. p. 148–62.

    Google Scholar 

  29. Rademacher F, Sachweh S, Zündorf A. A modeling method for systematic architecture reconstruction of microservice-based software systems. Berlin: Springer; 2020. p. 311–26.

    Google Scholar 

  30. Fraiwan M, Khasawneh N. A review of chatgpt applications in education, marketing, software engineering, and healthcare: Benefits, drawbacks, and research directions. 2023. arXiv Preprint. arXiv:2305.00237.

  31. Brown T, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877–901.

    Google Scholar 

  32. Ray PP. ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet Things Cyber Phys Syst. 2023.

  33. Sobania D, Briesch M, Hanna C, Petke J. An analysis of the automatic bug fixing performance of chatGPT. Los Alamitos: IEEE Computer Society; 2023. p. 23–30. https://doi.ieeecomputersociety.org/10.1109/APR59189.2023.00012.

  34. Haque MA, Li S. The potential use of ChatGPT for debugging and bug fixing. EAI Endorsed Trans AI Robot. 2023;2:e4–e4.

    Article  Google Scholar 

  35. Liu C, Bao X, Zhang H, Zhang N, Hu H, Zhang X, Yan M. Improving chatgpt prompt for code generation. 2023. arXiv preprint arXiv:2305.08360.

  36. White J, Hays S, Fu Q, Spencer-Smith J, Schmidt DC. Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design. 2023. arXiv preprint arXiv:2303.07839.

  37. Kabir S, Udo-Imeh DN, Kou B, Zhang T. Who answers it better? an in-depth analysis of chatgpt and stack overflow answers to software engineering questions. 2023. arXiv preprint arXiv:2308.02312.

  38. Hörnemalm A. ChatGPT as a software development tool: the future of development. 2023.

  39. Stojanovic T, Lazarević SD. The application of ChatGPT for identification of microservices. E-bus Tech Conf Proc. 2023;3(1):99–105.

    Google Scholar 

  40. Ahmad A, Waseem M, Liang P, Fahmideh M, Aktar MS, Mikkonen T. Towards human-bot collaborative software architecting with chatgpt. In: Proceedings of the 27th international conference on evaluation and assessment in software engineering. 2023. p. 279–85.

  41. White J, Fu Q, Hays S, Sandborn M, Olea C, Gilbert H, Elnashar A, Spencer-Smith J, Schmidt DC. A prompt pattern catalog to enhance prompt engineering with chatgpt. 2023. arXiv preprint arXiv:2302.11382.

  42. Abdelfattah AS, Rodriguez A, Walker A, Cerny T. Detecting semantic clones in microservices using components. SN Comput Sci. 2023;4:470.

  43. Zhou X, et al. Benchmarking microservice systems for software engineering research. In: Chaudron M, Crnkovic I, Chechik M, Harman M, editors. Proceedings of the 40th international conference on software engineering: companion proceedings, ICSE 2018, Gothenburg, Sweden, May 27–June 03, 2018. ACM; 2018. p. 323–324. https://doi.org/10.1145/3183440.3194991.

  44. Radford A, Narasimhan K, Salimans T, Sutskever I. Improving language understanding by generative pre-training. 2018.

  45. Friedman M. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc. 1937;32:675–701.

    Article  Google Scholar 

  46. Nemenyi PB. Distribution-free multiple comparisons. Princeton: Princeton University; 1963.

    Google Scholar 

  47. Conover WJ, Iman RL. Rank transformations as a bridge between parametric and nonparametric statistics. Am Stat. 1981;35:124–9.

    Article  Google Scholar 

  48. Elliott AC, Hynan LS. A SAS ®macro implementation of a multiple comparison post hoc test for a Kruskal–Wallis analysis. Comput Methods Progr Biomed. 2011;102:75–80.

    Article  Google Scholar 

  49. Hadi MU, Qureshi R, Shah A, Irfan M, Zafar A, Shaikh MB, Akhtar N, Wu J, Mirjalili. Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects. Authorea Preprints. 2023.

  50. Wohlin C, et al. Experimentation in software engineering. Berlin: Springer Science & Business Media; 2012.

    Book  Google Scholar 

  51. Ji Z, et al. Survey of hallucination in natural language generation. ACM Comput Surv. 2023;55:1–38.

    Article  Google Scholar 

  52. Liang T, et al. Encouraging divergent thinking in large language models through multi-agent debate; 2023. arXiv Preprint. arXiv:2305.19118.

  53. Kojima T, Gu SS, Reid M, Matsuo Y, Iwasawa Y. Large language models are zero-shot reasoners. Adv Neural Inf Process Syst. 2022;35:22199–213.

    Google Scholar 

  54. Shinn N, Labash B, Gopinath A. Reflexion: an autonomous agent with dynamic memory and self-reflection. 2023. arXiv preprint arXiv:2303.11366.

  55. Wang X, Wei J, Schuurmans D, Le Q, Chi E, Narang S, Chowdhery A, Zhou D. Self-consistency improves chain of thought reasoning in language models. 2022. arXiv preprint arXiv:2203.11171.

Download references

Funding

No funds, grants, or other support was received.

Author information

Author notes

  1. Ernesto Quevedo, Amr S. Abdelfattah, Alejandro Rodriguez, Jorge Yero and Tomas Cerny contributed equally to this work.

Authors and Affiliations

  1. Computer Science Department, Baylor University, Waco, USA

    Ernesto Quevedo, Amr S. Abdelfattah, Alejandro Rodriguez & Jorge Yero

  2. Systems and Industrial Engineering, University of Arizona, 1127 East James E Rogers Way #111, Tucson, AZ, 85721, USA

    Tomas Cerny

Authors
  1. Ernesto Quevedo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amr S. Abdelfattah
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alejandro Rodriguez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jorge Yero
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tomas Cerny
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Ernesto Quevedo: resources, conceptualization, methodology, software, data curation, formal analysis and investigation, writing-original draft preparation. Amr S. Abdelfattah: resources, formal analysis and investigation, data curation, validation, writing-original draft preparation. Alejandro Rodriguez: formal analysis and investigation, conceptualization, methodology, software, data curation, writing-original draft preparation, visualization, formal analysis and investigation. Jorge Yero: data curation, validation, writing-original draft preparation. Tomas Cerny: formal analysis and investigation, supervision, validation, writing-original draft preparation.

Corresponding author

Correspondence to Tomas Cerny.

Ethics declarations

Conflict of Interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Research Involving Human and/or Animals

Not applicable.

Informed Consent

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection “Recent Trends on Cloud Computing and Services Science” guest edited by Claus Pahl and Maarten van Steen.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Quevedo, E., Abdelfattah, A.S., Rodriguez, A. et al. Evaluating ChatGPT’s Proficiency in Understanding and Answering Microservice Architecture Queries Using Source Code Insights. SN COMPUT. SCI. 5, 422 (2024). https://doi.org/10.1007/s42979-024-02664-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-024-02664-0

Keywords

Search

Navigation