Appraisal and reporting of security assurance at operational systems level

Abstract

In this paper we discuss the issues relating the evaluation and reporting of security assurance of runtime systems. We first highlight the shortcomings of current initiatives in analyzing, evaluating and reporting security assurance information. Then, the paper proposes a set of metrics to help capture and foster a better understanding of the security posture of a system. Our security assurance metric and its reporting depend on whether or not the user of the system has a security background. The evaluation of such metrics is described through the use of theoretical criteria, a tool implementation and an application to a case study based on an insurance company network.

Introduction

The idea of producing Information Technology (IT) systems that are and remain 100% secure over time has long been dismissed as desirable but yet unachievable. This is partly due to both, the difficulty in anticipating during the system development process all the potential future threats to the system, and to the evolution of the systems’ environment parameters. Furnell (2009), in his article entitled ‘The Irreversible March of Technology’, remarks that although technology is perpetually evolving to the convenience of end-users and businesses, the security situation related to that technology has barely improved and the risks have escalated. In fact new technology often gives rise to newer ways of compromising IT systems’ security, thus adding extra layers of complexity to the endeavours to guarantee security.

A study conducted by Wool (2004), on firewall configurations, revealed that a common but sometimes overlooked source of security risks for large distributed and open IT systems is the improper deployment of the security mechanisms. In fact the security mechanisms, even properly elucidated during the risks assessment stage, may be deployed inappropriately or unidentified hazards in the system environment may render them less effective. How good, for instance, is a fortified door if the owner, inadvertently, leaves it unlocked? Or considering a more technical example, how relevant is a firewall for a critical system linked to the Internet if it is configured to allow any incoming packets? Therefore, monitoring and reporting on the security posture of IT systems must be carried out to determine compliance with security requirements (Jansen, 2009) and to gain assurance as to their ability to protect system assets adequately. This remains one of the fundamental tasks of security assurance which is here defined as the ground for confidence on deployed security mechanisms to meet their objectives. It is worth mentioning that our understanding of security assurance is in line with the Common Criteria's definition of assurance (Common Criteria Sponsoring Organizations, 2006).

Unfortunately, although assurance is a field that has been gaining momentum, partly due to the growing need for compliance within big corporations (Julisch, 2008), limited efforts have been dedicated to appraising security assurance of operational systems. Several reasons may explain this. First, security assurance was relegated to the shadow of security as both terms were often used interchangeably (Jelen and Williams, 1998). Hence, it has been assumed that addressing security also covers the assurance angle, when in reality there may be security protocols in place without evidence of them working properly (Jelen and Williams, 1998, Wool, 2004). The second reason is related to the target level of the security assurance analysis and/or evaluation. In fact most of the efforts have been dedicated to assurance at the software development level and at the end product criteria. Examples of such initiatives include: assurance cases (Strunk and Knight, 2006); UMLSec (Jürjens, 2005); Secure Tropos (Mouratidis and Giorgini, 2007a, Mouratidis and Giorgini, 2007b) and the Common Criteria or CC (Common Criteria Sponsoring Organizations, 2006). The rationale behind such efforts is that without a rigorous and effective way of dealing with security at system development process, the end product cannot be secure. While this is true, the emphasis on design and process evidence versus actual product software largely overshadows practical security concerns involving the implementation and deployment of operational systems (Jansen, 2009). Finally, published literature has focused on providing guidelines for identifying security metrics (Vaughn et al., 2002, Swanson et al., 2003, Seddigh et al., 2004, Savola, 2007) without providing indications on how to combine them into the quantitative or qualitative indicators that are important for a meaningful understanding of the security posture of an IT system component. The term security posture here refers to the status of the security mechanisms, i.e. whether there is any abnormally that may result in the system's security being jeopardized.

This paper aims to address such a gap in the current literature and support the evaluation of security assurance at the operational systems level, by introducing metrics for the continuous evaluation of security assurance of runtime systems. The outcome of evaluation is a statement about the extent to which confidence has been gained that the security mechanisms are operating properly. This paper considers a metric as a value, selected from a partially ordered set by some assessment process that represents an Information System related quality of some object of concern. It provides, or is used to create a description, prediction, or comparison, with some degree of confidence (WISSSR, 2001).

Approach and contribution: A prerequisite and challenge in seeking to evaluate security assurance of operational systems is to identify the keys characteristics of the security mechanisms and/or the system that ought to be assessed in order to develop assurance indicators. Only after that one can start to address how measurements of these characteristics can be integrated and communicated to users (including system administrators and security managers) to ensure a good understanding of the security posture. Such a challenge is partially answered by considering NIST's special publication NIST800-33 (Stoneburner, 2000). As a matter of fact, NIST asserts that assurance that the security objectives (integrity, availability, confidentiality, and accountability) will be adequately met by a specific implementation depends partly on whether the required security functionalities are present and correctly implemented. Consequently, probing the presence and correctness of the deployed security mechanisms in an operational system is paramount to gaining assurance. However, since assurance is about confidence, the quality of the mean of verification is as equally relevant and should be reflected in the assurance value. This is paramount since a clear correlation exists between the quality of the verification process and the reliability of the result achieved). Consider for instance the following scenario: one is using two different anti-viruses AV1 and AV2 for checking whether a system is currently immune of certain malware. Assuming one knows from previous use of AV1 that its effectiveness to detect malware is somehow dubious, whereas AV2 has proven to be very reliable in terms of malware detection precision. If one was to use AV1 and then AV2, and the verification outcome is “no virus found”, one would certainly be relieved. However, one's confidence in the actual result will differ depending on whether they have used AV1 or AV2. In fact, the confidence in the system security posture reading after using AV1 will be lower compared to the use of AV2.

No metrics are worthwhile if the results of applying them cannot be effectively understood and security assurance metrics are no exception. In the security assurance realm there are actually two main stakeholders: the common user who resorts to the IT system for fulfilling his/her daily activity, with a limited or no understanding of security; and on the other hand, the security expert. While the common users purely need assurance to boost their confidence in the system they are using, assurance from the security expert perspective mainly is needed to support better management of systems or network security. Thus according to Skroch et al. (2000), at least the reporting of the security assurance level (SAL) to the common user and the security expert has to be such that it is meaningful and useful to each of them. To illustrate, the correctness information related to a security mechanism could be enough for a security expert to grasp the urgency of a situation while meaning little to the common user. For instance, saying to a user that his/her firewall configuration is such that it allows all incoming connection may be less meaningful than telling him the likelihood of his/her firewall protecting him/her against external attack is nearly null. Therefore it is pertinent to determine ways in which security (SA) information may be beneficially communicated to a user without much exposure to security. Providing meaningful security mechanisms without knowing the system and environments in which the component (system) could be deployed is a difficult task (Grunske, 2007). Similarly, SA is a context dependent concept. Consider the following example: Alice may feel very confident in using an unsecured wireless connection for simple Internet browsing but that confidence would drop considerably if Alice was to use it for Internet banking. We could state that purpose 1 (web browsing) requires low security or that its context security criticality level is low. This is due to the fact that any potential risk impact for that context will be relatively low for the user, whereas the context security criticality for purpose 2 (Internet Banking) is high. This example illustrates that a user without any security background can still make informed decisions on pursuing or not a course of action, if she/he is provided SALs based on the security posture and the security criticality of the context in which their system operates.

Based on the above analysis, the main contribution of the paper can be summarized as: the provision of metrics for appraising the SAL of an operational system’ security. Those metrics are meant to help users, with and without security expertise, to understand the security posture of their system. Such a SAL depends on:

• The quality of the software probe doing the verification. We developed a metric taxonomy based on the Common Criteria and the System Security Engineering Capability Maturity Model (SSE-CMM, 1999). In fact not knowing how good a verification process or a software probe is, would result in no rational way of judging the reliability of the measurements it provides. A probe is a software program inserted in a system for the purpose of monitoring or collecting data about its security.

• The reported status of the security mechanism at a given time (investigate the Availability and Conformity of the security mechanism). A security mechanism for which deployment does not conform to the policy specification is less assured to provide adequate protection to a system. The outcome from (i) is then used as a cap to the possible values that can be obtained by adopting a given probe or more generally a verification process. The idea behind such consideration is that one cannot provide more assurance than its quality.

• The context in which the system is operating: One's confidence in a safeguard to meet its objectives and protect a system will be higher in a usage context that has a low security criticality (therefore requiring a low level security) than in a high security critical context.

Results from (ii) mainly aim at users with security background; while outputs from (ii) and (iii) are integrated in a SA function (refer to Section 7) to yield a contextual value of SA more relevant to those users without security expertise.

Outline: The rest of the paper is organized as follows: Section 2 provides a general overview of existing SA evaluation approaches, as reported in the literature. In Section 3, the assurance evaluation model is linked to risk assessment process. In Section 4, we present the developed probe quality metric taxonomy while in Section 5 the SA checks are discussed. Section 6 describes the process of elucidating the security criticality of the context in which a system operates and Section 7 presents the contextual SA function. Section 8 illustrates the case study and discusses the results obtained from applying our approach. In Section 9, the agents’ organization adopted for the evaluation of the assurance is described, while Section 10 concludes the paper.