Skip to main content
SpringerLink
Log in

Taxonomy of quality metrics for assessing assurance of security correctness

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Assurance is commonly considered as “something said or done to inspire confidence” (Webster dictionary). However, the level of confidence inspired from a statement or an action depends on the quality of its source. Similarly, the assurance that the deployed security mechanisms exhibit an appropriate posture depends on the quality of the verification process adopted. This paper presents a novel taxonomy of quality metrics pertinent for gaining assurance in a security verification process. Inspired by the systems security engineering capability maturity model and the common criteria, we introduce five ordinal quality levels for a verification process aimed at probing the correctness of runtime security mechanisms. In addition, we analyse the mapping between the quality levels and different capability levels of the following verification metrics families: coverage, rigour, depth and independence of verification. The quality taxonomy is part of a framework for the Security Assurance of operational systems. These metrics can also be used for gaining assurance in other areas such as legal and safety compliance. Furthermore, the resulting metrics taxonomy could, by identifying appropriate quality security requirements, assist manufacturers of information technology (IT) in developing their products or systems. Additionally, the taxonomy could also empower consumers in IT security product selection to efficaciously and effectively match their organisational needs, while IT security evaluators can use it as a reference point when forming judgments about the quality of a security product. We demonstrate the applicability of the proposed taxonomy through access control examples.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Bagheri, E., & Gasevic, D. (2010). Assessing the maintainability of software product line feature models using structural metrics. Software Quality Journal, 19(3), 579–612.

    Article  Google Scholar 

  • Bellovin, S. M. (2006). On the brittleness of software and the infeasibility of security metrics. IEEE Security &Privacy, 4(4), 96.

    Article  Google Scholar 

  • Bodeau, D. (2001). Information assurance assessment: Lessons-learned and challenges. In Proceedings of WISSRR 2001, Williamsburg, VA.

  • Bulut, E., Khadraoui, D., & Marquet, B. (2007). Multi-agent based security assurance monitoring system for telecommunication infrastructures. In Proceedings of the communication, network, and information security conference, Berkeley, California. Anaheim, CA, USA: ACTA Press.

  • Chaula, J. A., Yngström, L., & Kowalski, S. (2005). Security metrics and evaluation of information systems security. In Proceedings of the 4th annual conference on information security for South Africa (pp. 1–11). Pretoria, South Africa: ISSA.

  • Fenton, N. E., Neil, M., Marsh, W., Hearty, P., Radlinski, L., & Krause, P. (2008). On the effectiveness of early life cycle defect prediction with Bayesian Nets. Empirical Software Engineering, 13(5), 499–537.

    Article  Google Scholar 

  • Fenton, N., & Pfleeger, S. L. (1998). Software metrics: A rigorous and practical approach (2nd ed.). Boston: PWS Publishing.

    Google Scholar 

  • Fong, E., Kass, M., Rhodes, T., & Boland, F. (2010). Structured assurance case methodology for assessing software trustworthiness. In Proceedings of the 2010 fourth international conference on secure software integration and reliability improvement companion (pp. 32–33). Singapore: IEEE Computer Society.

  • Furnell, S. M. (2009). The irreversible march of technology. Information Security Technical Report, 14(4), 176–180.

    Article  Google Scholar 

  • Goertzel, K. M., Winograd, T., McKinley, H. L., Oh, L. J., Colon, M., McGibbon, T., et al. (2007). Software security assurance: State of the art report. Available at: http://iac.dtic.mil/iatac/download/security.pdf. Accessed 10 May 2011.

  • Goodenough, J., Lipson, H., & Weinstock, C. (2008). Arguing security-creating security assurance cases. Available at https://buildsecurityin.us-cert.gov/bsi/articles/knowledge/assurance/643-BSI.html. Accessed 7 March 2011.

  • Grunske, L., & Joyce, D. (2008). Quantitative risk-based security prediction for component-based systems with explicitly modeled attack profiles. Journal of Systems and Software, 81(8), 1327–1345.

    Article  Google Scholar 

  • Hecker, A., & Riguidel, M. (2009). On the operational security assurance evaluation of networked IT systems. In Proceedings of the 9th international conference on smart spaces and next generation wired/wireless networking and second conference on smart spaces. Lecture Notes in Computer Science (Vol. 5764, pp. 266–278). Berlin, Heidelberg: Springer.

  • Hunter, R., & Out, D. J. (2005). Low assurance protection profile for a software based personal firewall for home internet use BSI-PP-0014. Available at: https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Zertifizierung/ReportePP/PP0014b_pdf.pdf?__blob=publicationFile. Accessed 15 March 2011.

  • ISO/IEC 15408. (2006a). Common criteria for information technology, part 1–3, version 3.1. Geneva, Switzerland: International Organisation for Standardization and the International Electro-technical Commission.

    Google Scholar 

  • ISO/IEC 15504–5. (2006b). Information technology process assessment part 4: Guidance on use for process improvement and process capability determination. Geneva, Switzerland: International Organisation for Standardization and the International Electro-technical Commission.

    Google Scholar 

  • ISO/IEC 15939. (2007). Systems and software engineering—Measurement process. Geneva, Switzerland: International Organisation for Standardization and the International Electro-technical Commission.

    Google Scholar 

  • ISO/IEC 21827. (2008). Information technology—systems security engineering—capability maturity model (SSE-CMM). Geneva, Switzerland: International Organisation for Standardization and the International Electro-technical Commission.

    Google Scholar 

  • Jansen, W. (2009). Directions in security metrics research. Gaithersburg, MD: National Institute of Standards and Technology Special publication# NISTIR 7564, NIST.

    Google Scholar 

  • Jelen, G. F, & Williams, J. R. (1998). A practical approach to measuring assurance. In Proceedings of the 14th annual computer security applications conference (ACSAC 98) (pp. 333–343). Phoenix, AZ: IEEE Xplore.

  • Julisch, K. (2008). Security compliance: The next frontier in security research. In Proceedings of the New Security Paradigms Workshop (pp. 71–74). New York: ACM.

  • Jürjens, J. (2005). Secure systems development with UML. Berlin: Springer.

    MATH  Google Scholar 

  • Kitchenham, B., Pfleeger, S., & Fenton, N. (1995). Towards a framework for software measurement validation. IEEE Transactions on Software Engineering, 21(12), 929–944.

    Article  Google Scholar 

  • Klevinsky, T. J., Laliberte, S. A., & Gupta, A. (2002). Hack I.T.—security through penetration testing. Boston, Massachusetts, USA: Addison-Wesley.

    Google Scholar 

  • Lee, J., Lee, J., Lee, S., & Choi, B. (2003). A CC-based security engineering process evaluation model. In Proceedings of the 27th annual international computer software and applications conference (COMPSAC’03) (pp. 130–135). Dallas: IEEE Xplore.

  • Lipow, M. (1982). Number of faults per line of code. IEEE Transactions on Software Engineering, 8(4), 437–439.

    Article  Google Scholar 

  • Manadhata, P. K., & Wing, J. M. (2010). An attack surface metric. IEEE Transactions on Software Engineering, (99).

  • Marquet, B., Dubus, S., & Blad, C. (2010). Security assurance profile for large and heterogeneous telecom and IT infrastructures. In Proceedings of the 7th international symposium on risk management and cyber-informatics (RMCI’10), Orlando, Florida, USA. http://www.iiis.org/CDs2010/CD2010SCI/RMCI_2010/PapersPdf/RA432SS.pdf. Accessed 15 March 2011.

  • Liang T., & Ming-Tian, Z. (2006). A new evaluation strategy based on combining CC and SSE-CMM for security systems and products. In Proceedings of 5th international conference on grid and cooperative computing (GCC’06) (pp. 395–403). Washington, DC: IEEE Computer Society.

  • Mouratidis, H., & Giorgini, P. (2007). Secure Tropos: A security-oriented extension of the Tropos methodology. International Journal of Software Engineering and Knowledge Engineering, 17(2), 285–309.

    Article  Google Scholar 

  • NASA. (2004). Software assurance standard, NASA technical standard, NASA-STD-8739.8 w/Change 1, National Aeronautics and Space Administration. Available at: http://www.hq.nasa.gov/office/codeq/doctree/87398.pdf. Accessed 15 March 2011.

  • Ouedraogo, M. (2011) Valuation and reporting of security assurance at operational systems level. PhD thesis, University of East London, England, UK.

  • Ouedraogo, M., Khadraoui, D., Mouratidis, H., & Dubois, E. (2011). Appraisal and reporting of security assurance at operational systems level. Journal of Software and Systems. doi:10.1016/j.jss.2011.08.013.

  • Pavlich-Mariscal, J. A., Demurjian, S. A., & Michel, L. D. (2010). A framework for security assurance of access control enforcement code. Computers & Security, 29(7), 770–784.

    Article  Google Scholar 

  • Payne, S. C. (2006). A guide to security metrics. SANS Institute InfoSec Reading Room. http://www.sans.org/reading_room/whitepapers/auditing/guide-security-metrics_55. Accessed 15 March 2011.

  • Pham, N., Baud, L., Bellot, P., & Riguidel, M. (2008). A near real-time system for security assurance assessment. In Proceedings of the 3rd international conference on internet monitoring and protection (pp. 152–160). Bucharest, Romania: IEEE Computer Society.

  • Rhodes, T., Boland, F., Fong, E., & Kass, M. (2010). Software assurance using structured assurance case models. Journal of Research of the National Institute of Standard and Technology, 115(3), 209–216.

    Article  Google Scholar 

  • Savola, R. M. (2007). Towards a taxonomy for information security metrics. In Proceedings of ACM workshop on quality of protection QOP’07 (pp. 28–30). New York: ACM.

  • Savola, R. M. (2010). On the feasibility of utilizing security metrics in software-intensive systems. International Journal of Computer Science and Network Security, 10(1), 230–239.

    Google Scholar 

  • Seddigh, N., Pieda, P., Matrawy, A., Nandy, B., Lambadaris, L. & Hatfield, A. (2004). Current trends and advances in information assurance metrics. In Proceedings of second annual conference on privacy, security and trust (PST’04) (pp. 197–205). Fredericton, Canada.

  • Sheyner, O. M. (2004). Scenario graphs and attack graphs. PhD thesis, School of Computer Science Carnegie Mellon University.

  • Skroch, M. J., McHugh, J., & Wiliams, J. M. (2000). Information assurance metrics: Prophecy, process, or pipedream? In Proceedings of national information system security conference, Baltimore, USA.

  • Stoneburner, G. (2001). Underlying technical models for information technology security. Gaithersburg, MD: National Institute of Standards and technology Special publication #800–33, NIST.

    Google Scholar 

  • Strunk, E. A., & Knight, J. C. (2008). The essential synthesis of problem frames and assurance cases. Experts Systems the Journal of Knowledge Engineering, 25(1), 9–27.

    Article  Google Scholar 

  • Swanson, M., Nadya, B., Sabato, J., Hash, J., & Graffo, L. (2003). Security metrics guide for information technology systems. Gaithersburg, MD: NIST Special publication #800–55, NIST.

    Google Scholar 

  • Van Lamsweerde, A. (2009). Requirements engineering: From system goals to UML models to software specifications. West Sussex, England: Wiley.

    Google Scholar 

  • Vaughn, R. B., Henning, R., & Siraj, A. (2003). Information assurance measures and metrics—state of practice and proposed taxonomy. In Proceedings of the IEEE/HICSS’03 (p. 331). Big Island, Hawaii: IEEE Computer Society.

  • Williams, J. R., Schaefer, M., & Landoll, D. J. (1995). Pretty good assurance. In Proceedings of new security paradigms workshop (p. 82). La Jolla, CA: IEEE Computer Society.

  • WISSRR Workshop on Information, Security System Scoring and Ranking. (2001). Information system security attribute quantification or ordering (commonly but improperly know as security metrics). In Workshop proceedings, Williamsburg, VA, 21–23 May.

  • Wool, A. (2004). A quantitative study of firewall configuration errors. IEEE Computer, 37(6), 62–67.

    Article  Google Scholar 

  • Zuccato, A., Marquet, B., Papillon, S., & Alden, M. (2006). Service oriented modelling of communication infrastructure for assurance. In Proceedings of IEEE Information Assurance Workshop (pp 1–8). West Point: IEEE Xplore.

Download references

Acknowledgments

This work has been supported by the TITAN project and financed by the national fund of research of the Grand Duchy of Luxembourg under contract C08/iS/21.

Author information

Authors and Affiliations

  1. Service Science and Innovation Department (SSI), Public Research Center Henri Tudor, 1855, Kirchberg, Luxembourg

    Moussa Ouedraogo, Djamel Khadraoui & Eric Dubois

  2. School of Architecture, Computing and Engineering, University of East London, London, UK

    Moussa Ouedraogo, Haralambos Mouratidis & David Preston

  3. VTT Technical Research Centre of Finland, Oulu, Finland

    Reijo M. Savola

Authors
  1. Moussa Ouedraogo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Reijo M. Savola
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haralambos Mouratidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Preston
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Djamel Khadraoui
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Eric Dubois
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Moussa Ouedraogo.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ouedraogo, M., Savola, R.M., Mouratidis, H. et al. Taxonomy of quality metrics for assessing assurance of security correctness. Software Qual J 21, 67–97 (2013). https://doi.org/10.1007/s11219-011-9169-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-011-9169-0

Keywords

Search

Navigation