Skip to main content
Springer Nature Link
Log in

Generative Artificial Intelligence for the Visualization of Source Code as Comics

  • Conference paper
  • First Online:
Human Interface and the Management of Information (HCII 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14690))

Included in the following conference series:

Abstract

Data comics offer an innovative and accessible approach to visualizing abstract data, like source code. However, creating these comics is very challenging, as it requires an artist who can conceive and draw the comic while having a deep knowledge of the abstract data. This work explores the application of state-of-the art generative AI models, specifically GPT-4 and DALL\(\cdot \)E 3, to generate a complete comic using a zero-shot approach with three different prompts. Our experiment focuses on generating comics from Python source code. Through a qualitative evaluation, we observed that chain-of-thought prompting could enhance the quality of the generated comics, showcasing the potential advantages and limitations of current generative AI models in creating comics aimed at software comprehension.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ahmad, A., Waseem, M., Liang, P., Fahmideh, M., Aktar, M.S., Mikkonen, T.: Towards human-bot collaborative software architecting with ChatGPT. In: Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering. pp. 279–285 (2023)

    Google Scholar 

  2. Bach, B., Kerracher, N., Hall, K.W., Carpendale, S., Kennedy, J., Henry Riche, N.: Telling stories about dynamic networks with graph comics. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 3670–3682 (2016)

    Google Scholar 

  3. Bach, B., Riche, N.H., Carpendale, S., Pfister, H.: The emerging genre of data comics. IEEE Comput. Graphics Appl. 37(3), 6–13 (2017)

    Article  Google Scholar 

  4. Bettin, B., Jarvie-Eggart, M., Steelman, K.S., Wallace, C.: Developing a comic-creation assignment and rubric for teaching and assessing algorithmic concepts. In: 2021 IEEE Frontiers in Education Conference (FIE), pp. 1–5. IEEE (2021)

    Google Scholar 

  5. Chotisarn, N., Merino, L., Zheng, X., Lonapalawong, S., Zhang, T., Xu, M., Chen, W.: A systematic literature review of modern software visualization. J. Visualization 23, 539–558 (2020)

    Article  Google Scholar 

  6. Dang, H., Goller, S., Lehmann, F., Buschek, D.: Choice over control: how users write with large language models using diegetic and non-diegetic prompting. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–17 (2023)

    Google Scholar 

  7. Dehouche, N., Dehouche, K.: What’s in a text-to-image prompt? The potential of stable diffusion in visual arts education. Heliyon (2023)

    Google Scholar 

  8. Fan, Z., Gao, X., Mirchev, M., Roychoudhury, A., Tan, S.H.: Automated repair of programs from large language models. In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1469–1481. IEEE (2023)

    Google Scholar 

  9. Farmer, L.S.: Information architecture and the comic arts: knowledge structure and access. In: Web Design and Development: Concepts, Methodologies, Tools, and Applications, pp. 569–588. IGI Global (2016)

    Google Scholar 

  10. Gozalo-Brizuela, R., Garrido-Merchan, E.C.: ChatGPT is not all you need. a state of the art review of large generative AI models. arXiv preprint arXiv:2301.04655 (2023)

  11. Hagberg, A., Schult, D., Swart, P.: NetworkX (2024). https://github.com/networkx/networkx

  12. Hagberg, A.A., Schult, D.A., Swart, P.J.: Exploring network structure, dynamics, and function using NetworkX. In: Varoquaux, G., Vaught, T., Millman, J. (eds.) Proceedings of the 7th Python in Science Conference, pp. 11 – 15. Pasadena, CA USA (2008)

    Google Scholar 

  13. He, Y., Cao, S., Shi, Y., Chen, Q., Xu, K., Cao, N.: Leveraging large models for crafting narrative visualization: a survey. arXiv preprint arXiv:2401.14010 (2024)

  14. Heidrich, D., Meinecke, A., Schreiber, A., Byška, J., Jänicke, S., Schmidt, J.: Towards a collaborative experimental environment for graph visualization research in virtual reality. In: EuroVis 2021-Posters (2021)

    Google Scholar 

  15. Heidrich, D., Schreiber, A.: Visualizing source code as comics using generative AI. In: 2023 Working Conference on Software Visualization (VISSOFT), pp. 40–44. IEEE (2023). https://doi.org/10.1109/VISSOFT60811.2023.00014

  16. Heidrich, D., Schreiber, A., Oberdörfer, S.: Towards generating labeled property graphs for comprehending c#-based software projects. In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pp. 1–4 (2022)

    Google Scholar 

  17. Herman, I., Melançon, G., Marshall, M.S.: Graph visualization and navigation in information visualization: a survey. IEEE Trans. Visual Comput. Graphics 6(1), 24–43 (2000)

    Article  Google Scholar 

  18. Jebb, A.T., Ng, V., Tay, L.: A review of key likert scale development advances: 1995–2019. Front. Psychol. 12 (2021). https://doi.org/10.3389/fpsyg.2021.637547

  19. Jeong, H., Kwon, G., Ye, J.C.: Zero-shot generation of coherent storybook from plain text story using diffusion models. arXiv preprint arXiv:2302.03900 (2023)

  20. Tomihisa, K., Satoru, K.: A general framework for visualizing abstract objects and relations (1991)

    Google Scholar 

  21. Kazemitabaar, M., Hou, X., Henley, A., Ericson, B.J., Weintrop, D., Grossman, T.: How novices use llm-based code generators to solve cs1 coding tasks in a self-paced learning environment. arXiv preprint arXiv:2309.14049 (2023)

  22. Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. Adv. Neural. Inf. Process. Syst. 35, 22199–22213 (2022)

    Google Scholar 

  23. Li, R., et al.: Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023)

  24. Liew, A., Mueller, K.: Using large language models to generate engaging captions for data visualizations. arXiv preprint arXiv:2212.14047 (2022)

  25. Likert, R.: A technique for the measurement of attitudes. Arch. Psychol. 22, 1–55 (1932)

    Google Scholar 

  26. Lin, Y., Xian, X., Shi, Y., Lin, L.: Mirrordiffusion: stabilizing diffusion process in zero-shot image translation by prompts redescription and beyond. IEEE Signal Process. Lett. (2024)

    Google Scholar 

  27. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9), 1–35 (2023)

    Article  Google Scholar 

  28. Maslov, S., Sneppen, K.: Specificity and stability in topology of protein networks. Science 296(5569), 910–913 (2002)

    Article  Google Scholar 

  29. McIntosh, T.R., Susnjak, T., Liu, T., Watters, P., Halgamuge, M.N.: From google Gemini to OpenAI q* (q-star): a survey of reshaping the generative artificial intelligence (AI) research landscape (2023)

    Google Scholar 

  30. McNicol, S.: The potential of educational comics as a health information medium. Health Inf. Librar. J. 34(1), 20–31 (2017)

    Article  Google Scholar 

  31. Melzi, S., Peñaloza, R., Raganato, A.: Does stable diffusion dream of electric sheep? (2023)

    Google Scholar 

  32. Misiak, M., Schreiber, A., Fuhrmann, A., Zur, S., Seider, D., Nafeie, L.: Islandviz: a tool for visualizing modular software systems in virtual reality. In: 2018 IEEE Working Conference on Software Visualization (VISSOFT), pp. 112–116. IEEE (2018)

    Google Scholar 

  33. OpenAI: GPT-4 technical report. arxiv 2303.08774 2, 13 (2023)

    Google Scholar 

  34. Ozkaya, I.: Application of large language models to software engineering tasks: opportunities, risks, and implications. IEEE Softw. 40(3), 4–8 (2023)

    Article  Google Scholar 

  35. Pavlichenko, N., Ustalov, D.: Best prompts for text-to-image models and how to find them. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2067–2071 (2023)

    Google Scholar 

  36. al Qaimari, G., Paton, N.W., Kilgour, A.C.: Visualizing advanced data modelling constructs. Inf. Software Technol. 36(10), 597–605 (1994)

    Article  Google Scholar 

  37. Qin, J., et al.: Diffusiongpt: LLM-driven text-to-image generation system. arXiv preprint arXiv:2401.10061 (2024)

  38. Raquel Navarro-Prieto, J.C.: Mental representation and imagery in program comprehension. In: Annual Workshop of the Psychology of Programming Interest Group (1999)

    Google Scholar 

  39. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)

    Google Scholar 

  40. Rothwell, E., Cheek-O’Donnell, S., Johnson, E., Wilson, A., Anderson, R.A., Botkin, J.: Exploring the use of a comic for education about expanded carrier screening among a diverse group of mothers. J. Commun. Healthc. 14(3), 252–258 (2021)

    Article  Google Scholar 

  41. Schäfer, M., Nadi, S., Eghbali, A., Tip, F.: An empirical evaluation of using large language models for automated unit test generation. IEEE Trans. Software Eng. 50, 85–105 (2023)

    Article  Google Scholar 

  42. Schreiber, A., Struminksi, R.: Visualizing the provenance of personal data using comics. Computers 7(1), 12 (2018)

    Article  Google Scholar 

  43. Siregar, H.F., Siregar, Y.H., Melani, M.: Perancangan aplikasi komik hadist berbasis multimedia. (JurTI) Jurnal Teknologi Informasi 2(2), 113–121 (2018)

    Google Scholar 

  44. Sobania, D., Briesch, M., Hanna, C., Petke, J.: An analysis of the automatic bug fixing performance of chatgpt. arXiv preprint arXiv:2301.08653 (2023)

  45. Sporns, O., Zwi, J.D.: The small world of the cerebral cortex. Neuroinformatics 2, 145–162 (2004)

    Article  Google Scholar 

  46. Suh, S.: Codetoon: a new visual programming environment using comics for teaching and learning programming. In: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 2. SIGCSE 2022, New York, NY, USA, p. 1177. Association for Computing Machinery (2022). https://doi.org/10.1145/3478432.3499254

  47. Suh, S., Lamorea, S., Law, E., Zhang-Kennedy, L.: Privacytoon: concept-driven storytelling with creativity support for privacy concepts. In: Designing Interactive Systems Conference, pp. 41–57 (2022)

    Google Scholar 

  48. Suh, S., Latulipe, C., Lee, K.J., Cheng, B., Law, E.: Using comics to introduce and reinforce programming concepts in CS1. In: SIGCSE, pp. 369–375 (2021)

    Google Scholar 

  49. Suh, S., Lee, M., Xia, G., law, E.: Coding strip: a pedagogical tool for teaching and learning programming concepts through comics. In: 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pp. 1–10 (2020). https://doi.org/10.1109/VL/HCC50065.2020.9127262

  50. Theis, S., Jentzsch, S., Deligiannaki, F., Berro, C., Raulf, A.P., Bruder, C.: Requirements for explainability and acceptance of artificial intelligence in collaborative work. In: Degen, H., Ntoa, S. (eds.) HCII 2023. LNCS, vol. 14050, pp. 355–380. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-35891-3_22

    Chapter  Google Scholar 

  51. Touvron, H., et al.: Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

  52. Wang, Z., Romat, H., Chevalier, F., Riche, N.H., Murray-Rust, D., Bach, B.: Interactive data comics. IEEE Trans. Visual Comput. Graphics 28(1), 944–954 (2022). https://doi.org/10.1109/TVCG.2021.3114849

    Article  Google Scholar 

  53. Wei, J., et al.: Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022)

  54. Wong, M.F., Guo, S., Hang, C.N., Ho, S.W., Tan, C.W.: Natural language generation and understanding of big code for AI-assisted programming: a review. Entropy 25(6), 888 (2023)

    Article  Google Scholar 

  55. Yu, H.: Conceptual art or readable contract: the use of comics in technical communication. Tech. Commun. Q. 29(3), 222–239 (2020)

    Article  Google Scholar 

  56. Zhao, Z., Wallace, E., Feng, S., Klein, D., Singh, S.: Calibrate before use: improving few-shot performance of language models. In: International Conference on Machine Learning, pp. 12697–12706. PMLR (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. German Aerospace Center (DLR), Institute for Software Technology, Münchener Straße 20, 82234, Weßling, Germany

    David Heidrich

  2. German Aerospace Center (DLR), Institute for Software Technology, Linder Höhe, 51147, Cologne, Germany

    Andreas Schreiber & Sabine Theis

Authors
  1. David Heidrich
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andreas Schreiber
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sabine Theis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Heidrich .

Editor information

Editors and Affiliations

  1. Tokyo City University, Tokyo, Japan

    Hirohiko Mori

  2. Tokyo University of Science, Tokyo, Japan

    Yumi Asahi

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Heidrich, D., Schreiber, A., Theis, S. (2024). Generative Artificial Intelligence for the Visualization of Source Code as Comics. In: Mori, H., Asahi, Y. (eds) Human Interface and the Management of Information. HCII 2024. Lecture Notes in Computer Science, vol 14690. Springer, Cham. https://doi.org/10.1007/978-3-031-60114-9_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-60114-9_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-60113-2

  • Online ISBN: 978-3-031-60114-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics