Abstract
Real-time feedback is very important, yet challenging to provide for free-text learner contributions in Technology-Enhanced Learning. We study whether a generic NLP pipeline can identify completeness features of learner ideas during security training. We apply PoS Tagging and Dependency Parsing on contextualised short texts, collected within a dedicated learning environment and we compare the results to an expert-annotated ground truth. We scan these contributions for the absence of responsible stakeholder (who) or featured action (how). A total of 1174 contributions in two security domains were analysed. We report precision on who (\(PPV=0.929\)) and on how (\(PPV=0.691\)). We consider the first result to be sufficient to provide real-time formative feedback for the case of absent who. Our results suggest that for the purposes of providing feedback in free input problem-solving exercises, generic transformer pipelines without fine-tuning can achieve good performance on stakeholder identification.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Pipeline documentation at: https://spacy.io/models#design-trf.
- 2.
Full dataset and heuristics code at: https://cco.works/opendata.
References
Ahmed, S., Ahmed, A., Eisty, N.U.: Automatic transformation of natural to unified modeling language: a systematic review. In: 2022 IEEE/ACIS 20th International Conference on Software Engineering Research, Management and Applications (SERA). IEEE (2022). https://doi.org/10.1109/sera54885.2022.9806783
Bagaria, V., Badve, M., Beldar, M., Ghane, S.: An intelligent system for evaluation of descriptive answers. In: 2020 3rd International Conference on Intelligent Sustainable Systems (ICISS), pp. 19–24 (2020). https://doi.org/10.1109/ICISS49785.2020.9316110
Dean, D.L., Hender, J.M., Rodgers, T.L., Santanen, E.L.: Identifying quality, novel, and creative ideas: constructs and scales for idea evaluation. J. Assoc. Inf. Syst. 7(1), 646–699 (2006). https://doi.org/10.17705/1jais.00106
Ekblom, P.: Crime Prevention, Security and Community Safety Using the 5Is Framework. Crime Prevention and Security Management. Palgrave Macmillan, UK (2010). https://doi.org/10.1057/9780230298996
Gao, P., et al.: Enabling efficient cyber threat hunting with cyber threat intelligence. In: 2021 IEEE 37th International Conference on Data Engineering (ICDE), pp. 193–204 (2021). https://doi.org/10.1109/ICDE51399.2021.00024
Gupta, S., Nishu, K.: Mapping local news coverage: precise location extraction in textual news content using fine-tuned BERT based language model. In: Proceedings of the 4th Workshop on Natural Language Processing and Computational Social Science, pp. 155–162. ACL, Online (2020). https://doi.org/10.18653/v1/2020.nlpcss-1.17
Kalia, A.K., Batta, R., Xiao, J., Vukovic, M.: Ensemble of unsupervised parametric and non-parametric techniques to discover change actions. In: 2021 IEEE 14th International Conference on Cloud Computing (CLOUD), pp. 572–577 (2021). https://doi.org/10.1109/CLOUD53861.2021.00074
Maya, A., Nazura, J., Muralidhara, B.L.: Recent trends in answer script evaluation - a literature survey. In: Proceedings of the 3rd International Conference on Integrated Intelligent Computing Communication & Security, ICIIC 2021, pp. 105–112. Atlantis Press (2021). https://doi.org/10.2991/ahis.k.210913.014
Ruskov, M., Celdran, J.M., Ekblom, P., Sasse, M.A.: Unlocking the next level of crime prevention: development of a game prototype to teach the conjunction of criminal opportunity. Inf. Technol. Control 10(3), 15–21 (2012). http://www.acad.bg/rismim/itc/sub/archiv/Paper3_3_2012.pdf
Ruskov, M., Ekblom, P., Sasse, M.A.: Getting users smart quick about security: results from 90 minutes of using a persuasive toolkit for facilitating information security problem solving by non-professionals (2022). https://doi.org/10.48550/ARXIV.2209.02420
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ruskov, M. (2023). Who and How: Using Sentence-Level NLP to Evaluate Idea Completeness. In: Wang, N., Rebolledo-Mendez, G., Dimitrova, V., Matsuda, N., Santos, O.C. (eds) Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky. AIED 2023. Communications in Computer and Information Science, vol 1831. Springer, Cham. https://doi.org/10.1007/978-3-031-36336-8_44
Download citation
DOI: https://doi.org/10.1007/978-3-031-36336-8_44
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36335-1
Online ISBN: 978-3-031-36336-8
eBook Packages: Computer ScienceComputer Science (R0)