Skip to main content
Springer Nature Link
Log in

Measuring the Effectiveness of U.S. Government Security Awareness Programs: A Mixed-Methods Study

  • Conference paper
  • First Online:
HCI in Business, Government and Organizations (HCII 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14038))

Included in the following conference series:

  • 1198 Accesses

Abstract

The goal of organizational security awareness programs is to positively influence employee security behaviors. However, organizations may struggle to determine program effectiveness, often relying on training policy compliance metrics (e.g., training completion rates) rather than measuring actual impact. Few studies have begun to discover approaches and challenges to measuring security awareness program effectiveness within compliance-focused sectors such as the United States (U.S.) government. To address this gap, we conducted a mixed-methods research study that leveraged both focus group and survey methodologies centered on U.S. Government organizations. We discovered that organizations do indeed place emphasis on compliance metrics and are challenged in determining other ways to gauge success. Our results can inform guidance and other initiatives to aid organizations in measuring the effectiveness of their security awareness programs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Organization names are for illustrative purposes only and do not signify the organizations’ participation in the study.

References

  1. 106th Congress: S.900 - Gramm-Leach-Bliley Act (1999). https://www.congress.gov/bill/106th-congress/senate-bill/900

  2. 113th Congress: Federal information security modernization act of 2014. Pub. L. 113–283, 128 Stat. 3073 (2014). https://www.govinfo.gov/app/details/PLAW-113publ283

  3. Alshaikh, M., Maynard, S.B., Ahmad, A., Chang, S.: An exploratory study of current information security training and awareness practices in organizations. In: 51st Hawaii International Conference on System Sciences, pp. 5085–5094 (2018)

    Google Scholar 

  4. Bada, M., Sasse, M.A., Nurse, J.R.: Cyber security awareness campaigns: Why do they fail to change behaviour? (2019). https://arxiv.org/ftp/arxiv/papers/1901/1901.02672.pdf

  5. Chaudhary, S., Gkioulos, V., Katsikas, S.: Developing metrics to assess the effectiveness of cybersecurity awareness program. J. Cybersecur. 8(1), tyac006 (2022)

    Google Scholar 

  6. Corbin, J., Strauss, A.L.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, 4th edn. Sage, Thousand Oaks, CA (2015)

    Google Scholar 

  7. Department of Health and Human Services: The HIPAA privacy rule (2021). https://www.hhs.gov/hipaa/for-professionals/privacy/index.html

  8. European Union Agency for Cybersecurity (ENISA): The new user’s guide: how to raise information security awareness (en) (2010). https://www.enisa.europa.eu/publications/archive/copy_of_new-users-guide

  9. Fertig, T., Schütz, A.E., Weber, K.: Current issues of metrics for information security awareness. In: European Conference on Information Systems (2020)

    Google Scholar 

  10. Government of Canada: Directive on security management (2019). https://www.tbs-sct.canada.ca/pol/doc-eng.aspx?id=32611 &section=procedure &p=H

  11. Haney, J., Jacobs, J., Furman, S., Barrientos, F.: NISTIR 8420A approaches and challenges of federal cybersecurity awareness programs (2022). https://nvlpubs.nist.gov/nistpubs/ir/2022/NIST.IR.8420A.pdf

  12. Jaeger, L.: Information security awareness: literature review and integrative framework. In: 51st Hawaii International Conference on System Sciences, pp. 4703–4712 (2018)

    Google Scholar 

  13. Krueger, R.A., Casey, M.A.: Focus groups: a practical guide for applied research. Sage (2015)

    Google Scholar 

  14. Manifavas, C., Fysarakis, K., Rantos, K., Hatzivasilis, G.: Dynamic security awareness program evaluation. In: Proceedings of the 16th International Conference on Human-Computer Interaction, pp. 258–269 (2014)

    Google Scholar 

  15. McDonald, N., Schoenebeck, S., Forte, A.: Reliability and inter-rater reliability in qualitative research: norms and guidelines for CSCW and HCI practice. In: ACM on Human-Computer Interaction, p. 72 (2019)

    Google Scholar 

  16. Monahan, D.: Security awareness training: it’s not just for compliance (2014). https://www.enterprisemanagement.com/research/asset-free.php/2734/pre/Report-Summary--Security-Awareness-Training:-It’s-Not-Just-for-Compliance-pre

  17. Muronga, K., Herselman, M., Botha, A., Veiga, A.D.: An analysis of assessment approaches and maturity scales used for evaluation of information security and cybersecurity user awareness and training programs: a scoping review. In: 2019 Conference on Next Generation Computing Applications (NextComp), pp. 1–6 (2019)

    Google Scholar 

  18. National Institute of Standards and Technology: pre-draft call for comments: Building a cybersecurity and privacy awareness and training program (2021). https://csrc.nist.gov/publications/detail/sp/800-50/rev-1/draft

  19. National Institute of Standards and Technology: FISSEA - Federal Information Security Educators (2022). https://csrc.nist.gov/projects/fissea

  20. Office of Management and Budget: Circular a-130 managing information as a strategic resource (2106). https://www.whitehouse.gov/omb/information-for-agencies/circulars/

  21. Rahim, A., Hayani, N., Hamid, S., Kia, M.L.M., Shamshirband, S., Furnell, S.: A systematic review of approaches to assessing cybersecurity awareness. Kybernetes 44(4), 606–622 (2015)

    Article  Google Scholar 

  22. Rantos, K., Fysarakis, K., Manifavas, C.: How effective is your security awareness program? an evaluation methodology. Inf. Secur. J. Global Perspect. 21(6), 328–345 (2012)

    Article  Google Scholar 

  23. SANS: Security awareness maturity model (2018). https://www.sans.org/security-awareness-training/blog/security-awareness-maturity-model-kit

  24. SANS: 2021 SANS security awareness report: Managing human cyber risk (2021). https://www.sans.org/security-awareness-training/resources/reports/sareport-2021/

  25. SANS: SANS security awareness resources (2022). https://www.sans.org/security-awareness-training/resources/

  26. Spitzner, L.: Security awareness metrics - what to measure and how (2021). https://www.sans.org/blog/security-awareness-metrics-what-to-measure-and-how/

  27. State of Massachusetts: Title 201 CMR 17.00 - Standards for the protection of personal information of residents of the commonwealth (2017). https://casetext.com/regulation/code-of-massachusetts-regulations/department-201-cmr-office-of-consumer-affairs-and-business-regulation/

  28. Wilson, M., Hash, J.: NIST Special Publication 800–50 - Building an information technology security awareness program (2003). https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-50.pdf

Download references

Author information

Authors and Affiliations

  1. National Institute of Standards and Technology, Gaithersburg, MD, 20899, USA

    Jody L. Jacobs, Julie M. Haney & Susanne M. Furman

Authors
  1. Jody L. Jacobs
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Julie M. Haney
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Susanne M. Furman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jody L. Jacobs .

Editor information

Editors and Affiliations

  1. City University of Hong Kong, Kowloon, Hong Kong

    Fiona Nah

  2. City University of Hong Kong, Kowloon, Hong Kong

    Keng Siau

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jacobs, J.L., Haney, J.M., Furman, S.M. (2023). Measuring the Effectiveness of U.S. Government Security Awareness Programs: A Mixed-Methods Study. In: Nah, F., Siau, K. (eds) HCI in Business, Government and Organizations. HCII 2023. Lecture Notes in Computer Science, vol 14038. Springer, Cham. https://doi.org/10.1007/978-3-031-35969-9_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-35969-9_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-35968-2

  • Online ISBN: 978-3-031-35969-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics