skip to main content
10.1145/3587102.3588788acmconferencesArticle/Chapter ViewAbstractPublication PagesiticseConference Proceedingsconference-collections
research-article

Automatic Problem Generation for CTF-Style Assessments in IT Forensics Courses

Published: 30 June 2023 Publication History

Abstract

In this experience paper, we present an automated assessment and marking generation framework to create capture-the-flag (CTF) questions in the context of Information Technology (IT) Forensics. This allows educators to generate many randomised Virtual Hard Disk (VHD) and packet capture (PCAP) files with different forensic artefacts for each student suitable for assessment tasks in disk-based and network-based forensic courses, respectively. These files are then inscribed inside quizzes, which are constructively aligned to what students have learned in their lecture and tutorial classes. We replaced our invigilated closed-book end-of-semester exams with these open-book multiple-attempt non-invigilated in-semester quizzes. We also conducted a survey asking students about, how the designed quizzes (1) were aligned with (and covering) the promised course learning outcomes, (2) were run to address academic integrity concerns, and (3) helped students manage their stress once their final exams are replaced by the presented quizzes.

References

[1]
The sleuthkit autopsy, 2023. https://www.sleuthkit.org/autopsy/, Last accessed 20 Jan 2023.
[2]
Exam stress, 2023. https://www.concordia.ca/cunews/offices/provost/health/topics/stress-management/exam-stress.html, Last accessed 19 Jan 2023.
[3]
Exterro forensics toolkit (ftk), 2023. https://www.exterro.com/forensic-toolkit, Last accessed 20 Jan 2023.
[4]
Isc2 2022 cyberseurity workforce study, 2023. https://www.isc2.org/Research/Workforce-Study, Last accessed 20 Jan 2023.
[5]
Autohotkey, 2023. https://www.autohotkey.com/, Last accessed 19 Jan 2023.
[6]
Ansible, 2023. https://www.ansible.com/, Last accessed 19 Jan 2023.
[7]
Embedded answers (cloze) question type, 2023. https://docs.moodle.org/400/en/Embedded_Answers_(Cloze)_question_type, Last accessed 2 Aug 2022.
[8]
Core documentation, 2023. http://coreemu.github.io/core/, Last accessed 2 Aug 2022.
[9]
Damn vulnerable web application (dvwa), 2023. https://github.com/digininja/DVWA, Last accessed 19 Jan 2023.
[10]
Exiftool, 2023. https://exiftool.org/, Last accessed 20 Jan 2023.
[11]
Template designer documentation, 2023. https://jinja.palletsprojects.com/en/3.0.x/templates/, Last accessed 15 Aug 2022.
[12]
Moodle open source learning platform, 2023. https://docs.moodle.org, Last accessed 19 Jan 2023.
[13]
Regripper, 2023. https://brettshavers.com/entry/regripper, Last accessed 19 Jan 2023.
[14]
Virtualbox, 2023. https://www.VirtualBox.org/, Last accessed 19 Jan 2023.
[15]
Wireshark, 2023. https://www.wireshark.org/, Last accessed 16 Aug 2022.
[16]
Yaml ain't markup language (YAML?), 2023. https://www.yaml.org/, Last accessed 19 Jan 2023.
[17]
R. S. Alsawaier. The effect of gamification on motivation and engagement. The International Journal of Information and Learning Technology, 2018.
[18]
M. Benzi, G. Lagorio, and M. Ribaudo. Automatic challenge generation for hands-on cybersecurity training. In 2022 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pages 496--503. IEEE, 2022.
[19]
J. Burket, P. Chapman, T. Becker, C. Ganas, and D. Brumley. Automatic problem generation for {Capture-the-Flag} competitions. In 2015 USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE 15), 2015.
[20]
T. J. Burns, S. C. Rios, T. K. Jordan, Q. Gu, and T. Underwood. Analysis and exercises for engaging beginners in online CTF competitions for security education. In 2017 USENIX Workshop on Advances in Security Education (ASE 17), Vancouver, BC, Aug. 2017. USENIX Association. URL https://www.usenix.org/conference/ase17/workshop-program/presentation/burns.
[21]
R. A. Chetwyn and L. Erdodi. Cheat detection in cyber security capture the flag games-an automated cyber threat hunting approach. Proceedings of the 28th C&ESAR, page 175, 2021.
[22]
S. V. Cole. Impact of capture the flag (ctf)-style vs. traditional exercises in an introductory computer security class. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1, ITiCSE '22, page 470--476, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392013. URL https://doi.org/10.1145/3502718.3524806.
[23]
M. Ellis, L. Baum, K. Filer, and S. H. Edwards. Experience report: Exploring the use of ctf-based co-curricular instruction to increase student comfort and success in computing. In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1, ITiCSE '21, page 303--309, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450382144. URL https://doi.org/10.1145/3430665.3456376.
[24]
P. Hulin, A. Davis, R. Sridhar, A. Fasano, C. Gallagher, A. Sedlacek, T. Leek, and B. Dolan-Gavitt. {AutoCTF}: Creating diverse pwnables via automated bug injection. In 11th USENIX Workshop on Offensive Technologies (WOOT 17), 2017.
[25]
M. Knüpfer, T. Bierwirth, L. Stiemert, M. Schopp, S. Seeber, D. Pöhn, and P. Hillmann. Cyber taxi: A taxonomy of interactive cyber training and education systems. In G. Hatzivasilis and S. Ioannidis, editors, Model-driven Simulation and Training Environments for Cybersecurity, pages 3--21, Cham, 2020. Springer International Publishing. ISBN 978--3-030--62433-0.
[26]
K. Leune and S. J. Petrilli. Using capture-the-flag to enhance the effectiveness of cybersecurity education. In Proceedings of the 18th Annual Conference on Information Technology Education, SIGITE '17, page 47--52, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450351003. URL https://doi.org/10.1145/3125659.3125686.
[27]
J. Mirkovic and P. A. H. Peterson. Class Capture-the-Flag exercises. In 2014 USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE 14), San Diego, CA, Aug. 2014. USENIX Association. URL https://www.usenix.org/conference/3gse14/summit-program/presentation/mirkovic.
[28]
M. Ribaudo and A. Valenza. Semi-automatic generation of cybersecurity exercises: A preliminary proposal. In Proceedings of the 2nd ACM SIGSOFT International Workshop on Ensemble-Based Software Engineering for Modern Computing Platforms, pages 16--21, 2019.
[29]
Z. C. Schreuders, T. Shaw, M. Shan-A-Khuda, G. Ravichandran, J. Keighley, and M. Ordean. Security scenario generator ({ { { { {SecGen)} } } } }: A framework for generating randomly vulnerable rich-scenario {VMs} for learning computer security and hosting {CTF} events. In 2017 USENIX Workshop on Advances in Security Education (ASE 17), 2017.
[30]
V. vábensky, J. Vykopal, P. Celeda, K. Tkácik, and D. Popovic. Student assessment in cybersecurity training automated by pattern mining and clustering. Education and Information Technologies, pages 1--32, 2022.
[31]
J. Vykopal, V. ?vábenský, and E.-C. Chang. Benefits and pitfalls of using capture the flag games in university courses. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education, SIGCSE '20, page 752--758, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450367936. URL https://doi.org/10.1145/3328778.3366893.
[32]
J. Vykopal, P. Celeda, P. Seda, V. ?vábensky, and D. Tovarnák. Scalable learning environments for teaching cybersecurity hands-on. In 2021 IEEE Frontiers in Education Conference (FIE), pages 1--9. IEEE, 2021.
[33]
J. Vykopal, V. ?vábensky, P. Seda, and P. Celeda. Preventing cheating in hands-on lab assignments. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1, pages 78--84, 2022.
[34]
Z. Yan and L. Yang. Assessment as learning: Maximising opportunities for student learning and achievement. Routledge, 2021.

Cited By

View all
  • (2024)Diverging assessments: What, Why, and ExperiencesProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630832(1161-1167)Online publication date: 7-Mar-2024
  • (2024)ALAN: Assessment-as-Learning Authentic Tasks for NetworkingProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630774(853-859)Online publication date: 7-Mar-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ITiCSE 2023: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1
June 2023
694 pages
ISBN:9798400701382
DOI:10.1145/3587102
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 June 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. academic integrity
  2. automation
  3. constructive alignment
  4. randomisation
  5. stress

Qualifiers

  • Research-article

Funding Sources

  • Australian Council of Deans of Information & Communications Technology (ACDICT)

Conference

ITiCSE 2023
Sponsor:

Acceptance Rates

Overall Acceptance Rate 552 of 1,613 submissions, 34%

Upcoming Conference

ITiCSE '25
Innovation and Technology in Computer Science Education
June 27 - July 2, 2025
Nijmegen , Netherlands

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)63
  • Downloads (Last 6 weeks)8
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Diverging assessments: What, Why, and ExperiencesProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630832(1161-1167)Online publication date: 7-Mar-2024
  • (2024)ALAN: Assessment-as-Learning Authentic Tasks for NetworkingProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630774(853-859)Online publication date: 7-Mar-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media