research-article

A Vignette-based Method for Improving Cybersecurity Talent Management through Cyber Defense Competition Design

Author:

David H. TobeyAuthors Info & Claims

SIGMIS-CPR '15: Proceedings of the 2015 ACM SIGMIS Conference on Computers and People Research

Pages 31 - 39

https://doi.org/10.1145/2751957.2751963

Published: 04 June 2015 Publication History

Abstract

The preliminary findings are reported from a four-year study of cybersecurity competency assessment and development achieved through the design of cyber defense competitions. The first year of the study focused on identifying the abilities that should indicate aptitude to perform well in the areas of operational security testing and advanced threat response. A recently developed method for Job Performance Modeling (JPM) is applied which uses vignettes -- critical incident stories -- to guide the elicitation of a holistic description of mission-critical roles grounded in the latest tactics, techniques and protocols defining the current state-of-the-art, or ground truth, in cyber defense. Implications are drawn for design of scoring engines and achievement of game balance in cyber defense competitions as a talent management system.

References

[1]

Anderson, L.W., Krathwohl, D.R., and Bloom, B.S. A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Longman, New York, 2001.

[2]

Assante, M.J. and Tobey, D.H. Enhancing the cybersecurity workforce. IEEE IT Professional 13, (2011), 12--15.

Digital Library

[3]

Benner, P.E. From novice to expert: Excellence and power in clinical nursing practice. Addison-Wesley, Menlo Park, CA, 1984.

[4]

Bloom, B.S. Taxonomy of educational objectives: The classification of educational goals. Longmans, Green, New York, 1956.

[5]

Boje, D.M. The storytelling organization: A study of story performance in an office-supply firm. Administrative Science Quarterly 36, (1991), 106--126.

[6]

Boje, D.M. Stories of the storytelling organization: A postmodern analysis of Disney as Tamara-land. Academy of Management Journal 38, (1995), 997--1035.

[7]

Boje, D.M. Narrative Methods for Organizational and Communication Research. Sage Publications, London, 2001.

[8]

Boyatzis, R.E. The competent manager: A model for effective performance. Wiley, New York, 1982.

[9]

Brown, Collins, and Duguid. Situated cognition and the culture of learning. Educational Researcher 18, (1989), 32--42.

[10]

Campion, M.A., Fink, A.A., Ruggenberg, B.J., Carr, L., Phillips, G.M., and Odman, R.B. Doing competencies well: Best practices in competency modeling. Personnel Psychology 64, (2011), 225--262.

[11]

Charness, N. Expertise in chess: The balance between knowledge and search. In K.A. Ericsson and J. Smith, eds., Toward a general theory of expertise: Prospects and Limits. Cambridge University Press, Cambridge, 1991, 39--63.

[12]

Chin, C. and Brown, D.E. Learning in science: A comparison of deep and surface approaches. Journal of Research in Science Teaching 37, 2 (2000), 109--138.

[13]

Le Deist, F.D. and Winterton, J. What is competence? Human Resource Development International 8, (2005), 27--46.

[14]

Endsley, M.R. Toward a theory of situation awareness in dynamic systems. Human Factors: The Journal of the Human Factors and Ergonomics Society 37, (1995), 32--64.

[15]

Flanagan, J.C. The critical incident technique. Psychological Bulletin 51, (1954), 327--358.

[16]

Franke, V. Decision-making under uncertainty: Using case studies for teaching strategy in complex environments. Journal of Military and Strategic Studies 13, 2 (2011), 1--21.

[17]

Gandhi, R.A., Siy, H., and Wu, Y. Studying software vulnerabilities. Crosstalk: The Journal of Defense Software Engineering, (2010), 16--20.

[18]

Garris, R., Ahlers, R., and Driskell, J.E. Games, motivation, and learning: A research and practice model. Simulation & gaming 33, 4 (2002), 441--467.

[19]

Gompert, D.C. Heads We Win-the Cognitive Side of Counterinsurgency (COIN): RAND Counterinsurgency Study-Paper 1. RAND Corporation, Santa Monica, CA, 2007.

[20]

Goodenough, J., Lipson, H., and Weinstock, C. Arguing Security - Creating Security Assurance Cases. Build Security In, 2012.

[21]

Guilford, J.P. The structure of intellect. Psychological Bulletin; Psychological Bulletin 53, 4 (1956), 267.

[22]

Hoffman, L.J., Rosenberg, T., Dodge, R., and Ragsdale, D. Exploring a national cybersecurity exercise for universities. IEEE Security & Privacy 3, 5 (2005), 27--33.

Digital Library

[23]

Johansen, R. Get there early: Sensing the future to compete in the present. Berrett-Koehler Publishers, San Francisco, Calif., 2007.

[24]

Kelly, T.P. and McDermid, J.A. Safety case construction and reuse using patterns. Proceedings of 16th International Conference on Computer Safety, Reliability and Security (SAFECOMP'97), Springer (1997), 55--69.

[25]

Kelly, T.P. and McDermid, J.A. A systematic approach to safety case maintenance. Reliability Engineering & System Safety 71, 3 (2001), 271--284.

[26]

Kelly, T.P. and Weaver, R.A. The goal structuring notation-a safety argument notation. Proceedings of the dependable systems and networks 2004 workshop on assurance cases, (2004).

[27]

Kiili, K. Digital game-based learning: Towards an experiential gaming model. The Internet and Higher Education 8, 1 (2005), 13--24.

[28]

Klein. Sources of Power: How people make decisions. MIT Press, 1998.

[29]

Lave, J. and Wenger, E. Situated learning: Legitimate peripheral participation. Cambridge University Press, Cambridge, England, 1991.

[30]

De Long, D.W. Lost knowledge: Confronting the threat of an aging workforce. Oxford University Press, Oxford, 2004.

[31]

Mislevy, R.J., Steinberg, L.S., and Almond, R.G. On the structure of educational assessments. Measurement 1, (2003), 3--62.

[32]

O'Neil, L.R., Assante, M.J., and Tobey, D.H. Snart Grid Cybersecurity: Job Performance Model Report. National Technical Information Service, Alexandria, VA, 2012.

[33]

O'Neil, L.R., Assante, M.J., Tobey, D.H., et al. Developing secure power systems professional competence: Alignment and gaps in workforce development programs. National Technical Information Service, Alexandria, VA, 2013.

[34]

Phillips, V. Learning grammar with a joystick and math with a mouse. Huffington Post, 2013.

[35]

Prensky, M. Digital game-based learning. McGraw-Hill, New York, 2001.

Digital Library

[36]

Schepens, W.J. and James, J.R. Architecture of a cyber defense competition. Systems, Man and Cybernetics, 2003. IEEE International Conference on, (2003), 4300--4305.

[37]

Schepens, W.J., Ragsdale, D.J., Surdu, J.R., and Schafer, J. The Cyber Defense Exercise: An evaluation of the effectiveness of information assurance education. The Journal of Information Security 1, 2 (2002).

[38]

Schraagen, J.M. Task analysis. In The Cambridge handbook of expertise and expert performance. Cambridge University Press, Cambridge, UK, 2006, 185--201.

[39]

Seddigh, N., Pieda, P., Matrawy, A., Nandy, B., Lambadaris, I., and Hatfield, A. Current trends and advances in information assurance metrics. Proceedings of the Second Annual Conference on Privacy, Security and Trust, (2004), 197--205.

[40]

Smith, K., Shanteau, J., and Johnson, P.E., eds. Psychological investigations of competence in decision making. Cambridge University Press, Cambridge, UK, 2004.

[41]

Tobey, D.H. Narrative's Arrow: Story sequences and organizational trajectories in founding stories. Standing Conference on Management and Organizational Inquiry, (2007).

[42]

Tobey, D.H. A competency model of advanced threat response. ATR Working Group Report NBISE-ATR-11-02. National Board of Information Security Examiners, Idaho Falls, ID, 2011.

[43]

Tobey, D.H., Gandhi, R.A., Reiter-Palmon, R., Yankelevich, M., and Pabst, K. ADAPTS: An evidence-based cyberlearning network for accelerating proficiency. 2013.

[44]

Tobey, D.H., Pusey, P., and Burley, D. Engaging learners in cybersecurity careers: Lessons from the launch of the National Cyber League. ACM InRoads 5, 1 (2014), 53--56.

Digital Library

[45]

Tobey, D.H., Reiter-Palmon, R., and Callens, A. Predictive Performance Modeling: An innovative approach to defining critical competencies that distinguish levels of performance. OST Working Group Report. National Board of Information Security Examiners, Idaho Falls, ID, 2012.

[46]

Tyler, J.A. and Boje, D.M. Sorting the relationship of tacit knowledge to story and narrative knowing. In Handbook of Research on Knowledge-Intensive Organizations. Information Science Reference, 2008.

[47]

Vogel, J.J., Vogel, D.S., Cannon-Bowers, J., Bowers, C.A., Muse, K., and Wright, M. Computer gaming and interactive simulations for learning: A meta-analysis. Journal of Educational Computing Research 34, 3 (2006), 229--243.

[48]

Vygotsky, L.S. Thought and language. MIT Press, Cambridge, MA, 1966.

[49]

Weaver, R.A., McDermid, J.A., and Kelly, T.P. Software safety arguments: towards a systematic categorisation of evidence. Proceedings of the 20th International System Safety Conference (ISSC 2002), (2002).

[50]

Weick, K.E., Sutcliffe, K.M., and Obstfeld, D. Organizing and the process of sensemaking. Organization Science 16, 4 (2005), 409--421.

Digital Library

[51]

Welander, P. Cyber Security Advice From The Field: An Interview With Control Engineering, Michael Assante And Tim Conway Offer Security Suggestions For Plant Operators. Control Engineering, 2013.

[52]

White, G.B. and Williams, D. The collegiate cyber defense competition. Proceedings of the 9th Colloquium for Information Systems Security Education, (2005).

[53]

Wouters, P., van Nimwegen, C., van Oostendorp, H., and van der Spek, E.D. A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, (in press).

[54]

Wouters, P., van der Spek, E.D., and van Oostendorp, H. Current practices in serious game research: A review from a learning outcomes perspective. In T. Connolly, M. Stansfield and L. Boyle, eds., Games-based learning advancements for multi-sensory human computer interfaces techniques and effective practices. Information Science Reference, Hershey, PA, 2009, 232--250.

[55]

Zbylut, M.L. and Ward, J.N. Developing interpersonal abilties with interactive vignettes. 2004.

[56]

Homeland Security Advisory Council CyberSkills Task Force Report. U.S. Departrment of Homeland Security, Washington, D.C., 2012

Cited By

Sabin MMcCauley RMacKellar BKumar A(2024)Using Vignettes to Categorize Behaviors That Students Associate with Dispositions2024 IEEE Frontiers in Education Conference (FIE)10.1109/FIE61694.2024.10892902(1-9)Online publication date: 13-Oct-2024
https://doi.org/10.1109/FIE61694.2024.10892902
Zacharis AGavrila RPatsakis CIkonomou D(2023)AI-assisted Cyber Security Exercise Content Generation: Modeling a Cyber Conflict2023 15th International Conference on Cyber Conflict: Meeting Reality (CyCon)10.23919/CyCon58705.2023.10181930(217-238)Online publication date: 29-May-2023
https://doi.org/10.23919/CyCon58705.2023.10181930
McCauley RSabin MKumar AKiesler NMacKellar BRaj RImpagliazzo J(2023)Using Vignettes to Elicit Students' Understanding of Dispositions in Computing Education2023 IEEE Frontiers in Education Conference (FIE)10.1109/FIE58773.2023.10342915(1-5)Online publication date: 18-Oct-2023
https://doi.org/10.1109/FIE58773.2023.10342915
Show More Cited By

Index Terms

A Vignette-based Method for Improving Cybersecurity Talent Management through Cyber Defense Competition Design
1. Computing methodologies
  1. Modeling and simulation
    1. Model development and analysis
      1. Modeling methodologies
2. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Project and people management
        Project staffing
        Systems analysis and design

Recommendations

The Cybersecurity Competition Federation
SIGMIS-CPR '15: Proceedings of the 2015 ACM SIGMIS Conference on Computers and People Research

In a time of global crisis in cybersecurity, competitions and related activities are rapidly emerging to provide fun and engaging ways of developing and assessing cybersecurity knowledge and skills. However, there is no neutral organization that brings ...
The Core Cyber-Defense Knowledge, Skills, and Abilities That Cybersecurity Students Should Learn in School: Results from Interviews with Cybersecurity Professionals

Our cybersecurity workforce needs surpass our ability to meet them. These needs could be mitigated by developing relevant curricula that prioritize the knowledge, skills, and abilities (KSAs) most important to cybersecurity jobs. To identify the KSAs ...
Cybersecurity Competitions in Education: Engaging Learners through Improved Game Balance
SIGMIS-CPR '15: Proceedings of the 2015 ACM SIGMIS Conference on Computers and People Research

In this presentation we will present models and research findings related to characteristics of competitions including: game balance, engagement, frameworks, and difficulty.

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMIS-CPR '15: Proceedings of the 2015 ACM SIGMIS Conference on Computers and People Research

June 2015

176 pages

ISBN:9781450335577

DOI:10.1145/2751957

Conference Chairs:
Diana Burley
The George Washington University, USA
,
Indira R. Guzman
Trident University International, USA
,
Program Chairs:
Daniel P. Manson
California State Polytechnic University, USA
,
Leigh Ellen Potter
Griffith University, Australia

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMIS: ACM Special Interest Group on Management Information Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 June 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

U.S. Department of Homeland Security

Conference

SIGMIS-CPR '15

Sponsor:

SIGMIS

SIGMIS-CPR '15: 2015 Computers and People Research Conference

June 4 - 6, 2015

California, Newport Beach, USA

Acceptance Rates

SIGMIS-CPR '15 Paper Acceptance Rate 26 of 47 submissions, 55%;

Overall Acceptance Rate 300 of 480 submissions, 63%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
378
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)2

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sabin MMcCauley RMacKellar BKumar A(2024)Using Vignettes to Categorize Behaviors That Students Associate with Dispositions2024 IEEE Frontiers in Education Conference (FIE)10.1109/FIE61694.2024.10892902(1-9)Online publication date: 13-Oct-2024
https://doi.org/10.1109/FIE61694.2024.10892902
Zacharis AGavrila RPatsakis CIkonomou D(2023)AI-assisted Cyber Security Exercise Content Generation: Modeling a Cyber Conflict2023 15th International Conference on Cyber Conflict: Meeting Reality (CyCon)10.23919/CyCon58705.2023.10181930(217-238)Online publication date: 29-May-2023
https://doi.org/10.23919/CyCon58705.2023.10181930
McCauley RSabin MKumar AKiesler NMacKellar BRaj RImpagliazzo J(2023)Using Vignettes to Elicit Students' Understanding of Dispositions in Computing Education2023 IEEE Frontiers in Education Conference (FIE)10.1109/FIE58773.2023.10342915(1-5)Online publication date: 18-Oct-2023
https://doi.org/10.1109/FIE58773.2023.10342915
Zacharis APatsakis C(2023)AiCEF: an AI-assisted cyber exercise content generation framework using named entity recognitionInternational Journal of Information Security10.1007/s10207-023-00693-z22:5(1333-1354)Online publication date: 19-Apr-2023
https://doi.org/10.1007/s10207-023-00693-z
Alammari ASohaib OYounes S(2022)Developing and evaluating cybersecurity competencies for students in computing programsPeerJ Computer Science10.7717/peerj-cs.8278(e827)Online publication date: 17-Jan-2022
https://doi.org/10.7717/peerj-cs.827
Hai-Jew S(2022)Online Calling Cards and Professional Profiles in Cybersecurity From Social MediaResearch Anthology on Advancements in Cybersecurity Education10.4018/978-1-6684-3554-0.ch008(157-195)Online publication date: 2022
https://doi.org/10.4018/978-1-6684-3554-0.ch008
Mases SMaennel KToussaint MRosa V(2021)Success Factors for Designing a Cybersecurity Exercise on the Example of Incident Response2021 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW)10.1109/EuroSPW54576.2021.00033(259-268)Online publication date: Sep-2021
https://doi.org/10.1109/EuroSPW54576.2021.00033
Winter RRuiz Rde Franco Rosa FJino M(2020)Cyber Mission Operations: A Literature Review17th International Conference on Information Technology–New Generations (ITNG 2020)10.1007/978-3-030-43020-7_5(31-37)Online publication date: 12-May-2020
https://doi.org/10.1007/978-3-030-43020-7_5
Hai-Jew S(2019)Online Calling Cards and Professional Profiles in Cybersecurity From Social MediaGlobal Cyber Security Labor Shortage and International Business Risk10.4018/978-1-5225-5927-6.ch009(149-186)Online publication date: 2019
https://doi.org/10.4018/978-1-5225-5927-6.ch009
Thomas LBalders MCountney ZZhong CYao JXu C(2019)Cybersecurity Education: From Beginners to Advanced Players in Cybersecurity Competitions2019 IEEE International Conference on Intelligence and Security Informatics (ISI)10.1109/ISI.2019.8823310(149-151)Online publication date: Jul-2019
https://doi.org/10.1109/ISI.2019.8823310
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten