skip to main content
10.1145/3664476.3670446acmotherconferencesArticle/Chapter ViewAbstractPublication PagesaresConference Proceedingsconference-collections
research-article

Assessing the Impact of Large Language Models on Cybersecurity Education: A Study of ChatGPT's Influence on Student Performance

Published: 30 July 2024 Publication History

Abstract

The popularity of chatbots to facilitate day-to-day business, including students and their study exercises, is on the rise. This paper investigates the extent and effects on the academic performance of students that leverage such tools. While many other approaches are hypothesized and discussed, we measure empirically. We recorded and compared the performance of cybersecurity students in weekly exercises and final exams over a period of three years. This allows us to have three groups with varying degrees of ChatGPT influence, namely no access, uncontrolled access, and controlled access. In an anonymous survey, we found that approximately 80% of our students utilize ChatGPT during the weekly assignments in 2023. However, none of them indicated this on their submission, despite it being a mandatory requirement. Through statistical analysis of achieved points in our sample groups, we identified that students perform similarly on the weekly assignments. However, their performance on the final examination deteriorates.

References

[1]
Hervé Abdi 2007. Bonferroni and Šidák corrections for multiple comparisons. Encyclopedia of measurement and statistics 3, 01 (2007), 2007.
[2]
Marian Daun and Jennifer Brings. 2023. How ChatGPT Will Change Software Engineering Education. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1. 110–116.
[3]
Olive Jean Dunn. 1964. Multiple comparisons using rank sums. Technometrics 6, 3 (1964), 241–252.
[4]
Ernst & Young GmbH. 2024. EY-Studierendenstudie 2024: Künstliche Intelligenz. https://www.ey.com/de_de/news/2024/03/ey-studierendenstudie-2024-kuenstliche-intelligenz
[5]
Mehmet Firat. 2023. What ChatGPT means for universities: Perceptions of scholars and students. Journal of Applied Learning and Teaching 6, 1 (2023).
[6]
Simone Grassini. 2023. Shaping the future of education: exploring the potential and consequences of AI and ChatGPT in educational settings. Education Sciences 13, 7 (2023), 692.
[7]
Heike Hofmann, Karen Kafadar, and Hadley Wickham. 2011. Letter-value plots: Boxplots for large data. Technical Report. had.co.nz.
[8]
Gwo-Jen Hwang and Ching-Yi Chang. 2021. A review of opportunities and challenges of chatbots in education. Interactive Learning Environments (2021), 1–14.
[9]
William H Kruskal and W Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American statistical Association 47, 260 (1952), 583–621.
[10]
Zeyan Liu, Zijun Yao, Fengjun Li, and Bo Luo. 2023. Check Me If You Can: Detecting ChatGPT-Generated Academic Writing using CheckGPT. arXiv preprint arXiv:2306.05524 (2023).
[11]
Chung Kwan Lo. 2023. What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature. Education Sciences 13, 4 (2023). https://doi.org/10.3390/educsci13040410
[12]
Kamil Malinka, Martin Peresíni, Anton Firc, Ondrej Hujnak, and Filip Janus. 2023. On the educational impact of ChatGPT: Is Artificial Intelligence ready to obtain a university degree?. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1. 47–53.
[13]
Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50–60.
[14]
Md Mostafizer Rahman and Yutaka Watanobe. 2023. ChatGPT for education and research: Opportunities, threats, and strategies. Applied Sciences 13, 9 (2023), 5783.
[15]
Malik Sallam. 2023. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare, Vol. 11. MDPI, 887.
[16]
Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. 2024. The Science of Detecting LLM-Generated Text. Commun. ACM 67, 4 (2024), 50–59.
[17]
Olia Tsivitanidou and Andri Ioannou. 2021. Envisioned pedagogical uses of chatbots in higher education and perceived benefits and challenges. In International Conference on Human-Computer Interaction. Springer, 230–250.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ARES '24: Proceedings of the 19th International Conference on Availability, Reliability and Security
July 2024
2032 pages
ISBN:9798400717185
DOI:10.1145/3664476
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 July 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. ChatGPT
  2. Education
  3. Teaching

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ARES 2024

Acceptance Rates

Overall Acceptance Rate 228 of 451 submissions, 51%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 129
    Total Downloads
  • Downloads (Last 12 months)129
  • Downloads (Last 6 weeks)20
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media