Skip to main content
SpringerLink
Log in

Strengthening Cyber Security Education: Designing Robust Assessments for ChatGPT-Generated Answers

  • Conference paper
  • First Online:
Machine Learning for Cyber Security (ML4CS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14541))

Included in the following conference series:

  • 34 Accesses

Abstract

Cyber security education has become a hot topic in Australia and many OECD countries due to increasing job demands for cyber security professionals. Designing authentic cyber security assessment tasks is an ongoing challenge, especially in the context of ChatGPT and similar AI-generated content (AIGC) tools. Some early studies suggest that the risks of using ChatGPT tools can be mitigated, but these studies overlooked cyber security education. This paper addresses this gap in the literature, focusing on assessment design in cyber security education in the presence of ChatGPT. While existing research has examined the transition from in-person to online education and ChatGPT’s capabilities, our study emphasizes the assessment structure and pedagogical approaches related to cyber security education. We conducted a systematic analysis by creating questions with four distinct prompts, feeding them to ChatGPT, and analyzing the answers with statistical tools. Our findings highlight the significance of question types and fact-checking in ChatGPT’s responses. We propose practical recommendations to enhance cyber security assessment design when incorporating ChatGPT. Our recommendations include incorporating recent academic references, using long essay questions, and thorough fact-checking to ensure the integrity of assessments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/f/awesome-chatgpt-prompts.

References

  1. Al-Hawawreh, M., Aljuhani, A., Jararweh, Y.: ChatGPT for cybersecurity: practical applications, challenges, and future directions. Cluster Comput. 26, 1–16 (2023)

    Article  Google Scholar 

  2. Apple Koh, L.L., Tan, M.X., Pee, G.Y.M., Lee, C.H., Colla, M., Kwan, W.L.: Exploring fair and effective online electronic exam in place of in-person examinations during remote learning. In: Proceedings of the 2021 IEEE International Conference on Engineering, Technology & Education (TALE). pp. 01–07 (2021). https://doi.org/10.1109/TALE52509.2021.9678669

  3. Batten, L., Pan, L.: Teaching digital forensics to undergraduate students. IEEE Secur. Priv. 6(3), 54–56 (2008). https://doi.org/10.1109/MSP.2008.74

    Article  Google Scholar 

  4. Chia, Y.X., et al.: Sentiments analysis and feedback among three cohorts in learning software engineering modules. In: 2022 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), pp. 105–111 (2022). https://doi.org/10.1109/TALE54877.2022.00025

  5. Falah, A., Pan, L., Chen, F.: A quantitative approach to design special purpose systems to measure hacking skills. In: Proceedings of the 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE). pp. 54–61 (2018). https://doi.org/10.1109/TALE.2018.8615431

  6. Firat, M.: What chatgpt means for universities: Perceptions of scholars and students. J. Appl. Learn. Teach. 6(1), 57–63 (2023)

    Google Scholar 

  7. Jeblick, K., et al.: ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports. Eur. Radiol. (2022). https://doi.org/10.1007/s00330-023-10213-1

  8. Knapp, K.J., Maurer, C., Plachkinova, M.: Maintaining a cybersecurity curriculum: professional certifications as valuable guidance. J. Inf. Syst. Educ. 28(2), 101 (2017)

    Google Scholar 

  9. Lau, Y.M., Barros, R.J., Gottipati, S., Shim, K.J.: Gamified online industry learning platform for teaching of foundational computing skills. In: 2022 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), pp. 112–119 (2022). https://doi.org/10.1109/TALE54877.2022.00026

  10. Liang, Z.: Contextualizing introductory app development course for first-year engineering students. In: 2022 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), pp. 86–92 (2022). https://doi.org/10.1109/TALE54877.2022.00022

  11. Malinka, K., Peresíni, M., Firc, A., Hujnak, O., Janus, F.: On the educational impact of ChatGPT: is artificial intelligence ready to obtain a university degree? In: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education, vol. 1. pp. 47–53 (2023)

    Google Scholar 

  12. Manoharan, S.: On individualized online assessments in stem subjects. In: Proceedings of the 2021 IEEE International Conference on Engineering, Technology and Education (TALE), pp. 255–260 (2021). https://doi.org/10.1109/TALE52509.2021.9678631

  13. Mijwil, M., Aljanabi, M., Ali, A.H.: ChatGPT: exploring the role of cybersecurity in the protection of medical information. Mesop. J. Cybersecur. 2023, 18–21 (2023)

    Google Scholar 

  14. Nathwani, G., Shoaib, A., Shafi, A., Furukawa, T.A., Huy, N.T.: Impact of COVID-2019 on school attendance problems. J. Global Health. 11, 03084 (2021)

    Article  Google Scholar 

  15. Okey, O.D., Udo, E.U., Rosa, R.L., Rodríguez, D.Z., Kleinschmidt, J.H.: Investigating ChatGPT and cybersecurity: a perspective on topic modeling and sentiment analysis. Comput. Secur. 135, 103476 (2023)

    Article  Google Scholar 

  16. Pan, L., Hutchinson, D., Khan, N.: Towards a sustainable assessment strategy for digital forensic education and training. In: Proceedings of IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE) 2012, pp. H1A–13–H1A–18 (2012). https://doi.org/10.1109/TALE.2012.6360318

  17. Rowe, D.C., Lunt, B.M., Ekstrom, J.J.: The role of cyber-security in information technology education. In: Proceedings of the 2011 Conference on Information Technology Education, pp. 113–122 (2011)

    Google Scholar 

  18. Susnjak, T.: ChatGPT: the end of online exam integrity? (2022)

    Google Scholar 

  19. Yeadon, W., Inyang, O.O., Mizouri, A., Peach, A., Testrow, C.P.: The death of the short-form physics essay in the coming AI revolution. Phys. Educ. 58(3), 035027 (2023)

    Article  Google Scholar 

  20. Zhang, C., et al.: One small step for generative AI, one giant leap for AGI: A complete survey on ChatGPT in AIGC era. arXiv preprint arXiv:2304.06488 (2023)

  21. Zhang, L., Chen, T., Ma, Y., Niu, J.: An online learning help approach based on Q&A sites in programming course. In: Proceedings of the 2022 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), pp. 31–37 (2022). https://doi.org/10.1109/TALE54877.2022.00014

Download references

Author information

Authors and Affiliations

  1. Swinburne University of Technology, 3122, Hawthorn, VIC, Australia

    Andrew Plapp, Caslon Chua & Jun Zhang

  2. Deakin University, 3220, Geelong, VIC, Australia

    Jianzhang Wu & Lei Pan

  3. RMIT University, 3000, Melbourne, VIC, Australia

    Chao Chen

Authors
  1. Andrew Plapp
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianzhang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Caslon Chua
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrew Plapp .

Editor information

Editors and Affiliations

  1. University of Queensland, St Lucia, QLD, Australia

    Dan Dongseong Kim

  2. RMIT University, Melbourne, VIC, Australia

    Chao Chen

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Plapp, A., Wu, J., Pan, L., Chen, C., Chua, C., Zhang, J. (2024). Strengthening Cyber Security Education: Designing Robust Assessments for ChatGPT-Generated Answers. In: Kim, D.D., Chen, C. (eds) Machine Learning for Cyber Security. ML4CS 2023. Lecture Notes in Computer Science, vol 14541. Springer, Singapore. https://doi.org/10.1007/978-981-97-2458-1_2

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-2458-1_2

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-2457-4

  • Online ISBN: 978-981-97-2458-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation