Keywords

1 Introduction

Nowadays, the vulnerability of modern industrial and critical infrastructures is an issue that concerns the worldwide public. The annual report from the European Union Agency for Network and Information Security (ENISA), “ENISA Threat Landscape report for 2014” [3], which focuses on the reports analysis from a large number of security labs, concludes that the threats in 2014 have undergone significant developments. The majority of the threats and their trends in the area of cyber physical systems, mobile computing, cloud computing, trust infrastructure, big data, Internet of things/interconnected devices/smart environments and in the area of network virtualization and software defined networking (SDN) have seen a significant increase.

Laurent F. Carrel wrote in his book, [1], on the practice of crisis management that it is necessary to organize and implement exercises systematically:

“Education and training are central tools in order to be prepared for crises. The focus is on being mentally prepared for critical situations. By means of systematic further education, knowledge and experience of individuals, staff or bodies of a crisis concerning the behavior in a crisis, are maintained and improved. During crises, such behavior pays off in many ways, in particular when right from the start a smoothly running organisation, efficient management procedures and institutions are available.”

As it is understood, the need for training, good preparation and evaluation to address such situations is obvious and vital. In order for someone to gain knowledge coming out of training, appropriate exercises should be implemented in these industrial and critical infrastructures. The successful implementation of such exercises depends on an accurate planning. For this reason, the generic as well as the major steps of the implementation of an exercise should be setup carefully in order for the objectives of an exercise to be achieved.

Many governmental and private organizations have developed methodologies for the exercises’ implementation. Such methodologies are described in the International Organization for Standardization’s (I.S.O.) standard 22398 [6], the Federal Crisis Management Training (HERMES) [4, 5], the Guide of the Swedish Civil Contingencies Agency (MSB) [7, 8], and the Good Practice Guide on National Exercises of ENISA [2].

This paper focuses on the analysis and comparison of these methodologies, which are the most popular and well marketed, by presenting their characteristics and requirements. Based on the outcome of this comparison, this paper will provide some questions related to the requirements and specifications, which an infrastructure should be compliant with in order to participate in an exercise, and suggestions for further work.

2 State of the Art

Exercises are the adjunct treatment to defend and protect infrastructures from cyber-attacks. For the planning of such exercises, many organizations have developed methodologies in order to give generic instructions which will lead to a successful exercise implementation.

The exercise life-cycle is similar in all the methodologies, consisting of four phases – the differentiations in the various methodologies relate to how they are named. For the purpose of this paper, the life-cycle of a methodology consists of four phases:

  • Pre-planning phase: the organizer identifies the need and the objectives of an exercise.

  • Planning phase: the organizer staffs the planning team and recruits participants.

  • Conducting phase: the exercise takes place.

  • Evaluating phase: the evaluation of the exercise is performed.

It is suggested that an exercise should not be implemented only once. The organization which carried out the exercise should take into account the results and the outcomes of the evaluation in order to plan a better exercise in the future. So, the exercise life-cycle can be captured in a spiral scheme as shown in Fig. 1.

Fig. 1.
figure 1

Exercise life-cycle

Full size image

In addition, a cartography of the available Information Technology (IT) systems should also be performed first, if an organization wishes to implement an exercise. Based on this cartography, the organizer could then proceed with the first phase of an exercise, in which the need and the set up of the objectives of the exercise will be identified.

During the planning phase, the organizer recruits staff to form the exercise planning team and the participants in the exercise. This means that the organizer should assign responsibilities and tasks to the appropriate personnel, leading to the creation of roles that are necessary for a successful implementation of the exercise.

In the following sections, the essential parts of the most popular exercise methodologies are analyzed and compared.

2.1 Hermes

Hermes OEx is a French methodology to organize exercises, where OEx means “Organisation d’ Exercices”. Using Hermes, one can define the results, the procedure of how the results will be achieved and the roles of the participants. The activities and the results which are produced using this method are generic. The major characteristic of Hermes is that it is oriented towards results. The results are set up based on the objectives of the project. Hermes helps the exercise organizer not to implement activities which are not related with the project. The results lead to the appropriate processes and roles.

Hermes uses the definition of “tailoring”, [4], which reflects that the project modifies the Hermes OEx. Someone can use this method in order to organize either large and complex exercises or short and simple ones. Tailoring helps the irrelevant results not to be produced and the whole effort to be put into achieving the objectives.

Using the term tailoring, Hermes:

  • wipes out the irrelevant activities, results, roles and decision points,

  • controls the remaining activities, results and roles in order to achieve the project objectives, and

  • adds additional, if they are required and deemed necessary, activities, results and roles.

For the organization of an exercise, [5], specific roles should be assigned to assist in the achievement of the project objectives. For this reason, the proposed roles are as follows:

  • Exercise director: The exercise director guides all the areas that need to be managed in an exercise and ensures that the actions and reactions of the entire exercise management staff are coordinated.

  • Head of directing staff: The head of the directing staff stages the exercise and is responsible for executing the exercise script and for incorporating the exercise incidents. The head of directing staff works closely with the head of the contact office. The best suited person for this purpose is the scenario and exercise script officer.

  • Head of exercise logistics: The head of exercise logistics manages and coordinates all the logistical aspects of the exercise.

  • Head of observers: The head of observers manages and coordinates the observers and prepares the content of any debriefing immediately after the exercise.

  • Head of the contact office: The head of the contact office manages and coordinates the members of the contact office and ensures that the enquiries and requests of exercise participants are dealt with quickly and competently. The head of the contact office works closely with the head of directing staff when the goal is to develop unprepared dilemmas and exercise incidents and to confront the exercise participants with them.

  • Exercise participants: They are the members of the organizations, staffs or function holders of the staff who participate in the exercise.

In the guidelines of Hermes, the term “results” is used to describe the outcomes. These outcomes are related directly to the exercise by governing its objectives, such as the preliminary concept of the exercise and the detailed exercise concept.

2.2 MSB

The Swedish Civil Contingencies Agency’s MSB (Myndigheten för Samhällsskydd och Beredskap) has as its major objective to increase the security in systems which are crucial and implicitly connected with normal life. If these systems are attacked, then the impact on the society will be large. In order to keep and improve the security of those systems, it is necessary for all the industrial and critical infrastructures to test and evaluate them. These tests can be implemented through the exercises. The exercises based on MSB could be categorized in strategic and operational, based on the goal that a specific infrastructure would like to achieve, [7].

MSB gives guidelines on how to structure and evaluate an exercise, on good crisis management and on procedures for good decision-making. One objective of this methodology is that the implementation and the impact of the measures, which the infrastructure has adopted, should be evaluated and made clear if the infrastructure is ready to face emergencies and crises. Another objective of this methodology is to evaluate whether the goals which have been set up have been achieved totally, partially or not at all.

MSB evaluates whether certain goals have been achieved and shows the benefits and the outcomes of the exercise. Using the evaluation, MSB compares the observations with the outcomes and the goals. Based on the description of the objectives, a report is generated which captures if they are close to reality based on the outcomes of the exercise, [8].

In order for the MSB exercise methodology to be implemented successfully, its tasks should be distributed in different roles. The generic groups of roles are the participants, the exercise management and the evaluator. These roles should understand and be trained very well on the tasks they are responsible for and they have to know the distinction between the roles and their tasks.

The various tasks assigned to participants, exercise management and evaluators should be explained thoroughly in order to create a mutual understanding among the three groups for their respective assignments. These distinctions can be summarized on the following points:

  1. 1.

    It should be clear what someone can earn by participating in an exercise and what will be the outcomes after the exercise implementation.

  2. 2.

    The participants of the exercise should be recruited as soon as possible in order to be informed and well-trained based on their needs. The outcomes will show where it is necessary to improve the exercise team.

  3. 3.

    The exercise should be initiated on a position of shared knowledge and information.

  4. 4.

    The evaluation process should be clear.

  5. 5.

    The importance and the necessity of conducting an exercise and why these specific objectives have been chosen should be shown.

  6. 6.

    The working environment during the exercise should be well-structured in order for the participants to be able to cooperate and exchange information effectively.

The outcomes can be carried out by certain objectives, [8], which could be discrete or combined as follows:

  • Investigative (exploratory)

  • Needs-oriented (diagnostic)

  • Process-oriented

  • Results/outcome - oriented.

2.3 ENISA’s Good Practice Guide

The European Commission and the Member States have realized that the exercises can play a significant role in increasing the resilience of public e-Communications networks. The authorities that manage industrial and critical infrastructures should be aware of their weaknesses and vulnerabilities by participating in exercises. For this reason, the European Union Agency of Network and Information Security (ENISA) has created and published a good practice guide for the organization and implementation of an exercise, which can be assumed that it is a methodology [2].

Based on this guide, the organization of an exercise is related to the exercise life-cycle. When someone desires to organize an exercise based on this methodology, he should set up the needs and the resources. Based on these, he can decide the type of the exercise which he can prepare, and he has to choose a high-level scenario and identify the participants.

The major value of implementing an exercise is that someone can increase the resilience of the infrastructure. There are different types of exercises depending on the complexity, the size, the examining procedures and the scenario. For the planning and organizing of the exercise, the organizer should:

  • Define the measures and processes which will be tested by drilling a specific function, or by developing a business contingency plan.

  • Set which the target group will be.

  • Identify the resources available for planning and for conducting the exercise.

  • Set up the degree of commitment that can be expected from the participants.

  • Take into account any available previous experience from other exercises on the side of both the organizers and the participants.

The most important thing in order to organize an exercise is to identify the need of planning it. Using the exercise, someone can measure and evaluate the processes which the infrastructure follows when an incident happens. The organizers should focus on measures which need evaluation, review, improvement and training.

In the case of sectorial exercises or cross-sectorial exercises, the organizer should try to convince the participants to cooperate and coordinate among them focusing on testing the cooperation and showing any existing interdependencies. An organizer can focus on measures such as the common situational awareness of the participants, the needed collaboration (internally and externally) to address the problem, the coordination of resources, the logistics and support capabilities, etc.

Another significant goal for organizing an exercise is to create a high-level scenario which addresses the achievement of the objectives. There are two reasons for choosing the appropriate scenario:

  • It helps the organizer to recruit the most appropriate participants to assist in creating a suitable scenario upon the objectives.

  • It helps the organizers to have a plan of the exercise before they recruit the planners.

Based on the ENISA’s guide, [2], there are many different types of roles during the exercise. The major roles which an infrastructure should have are the following:

  • Organizer: The organizer is the organization that drives the process of exercise organization.

  • Planner: The planners are organizations or individuals who participate in the planning of the exercise.

  • Participant (also Participating Organization, or Participating Stakeholder): A participant is an organization or individual who will play a certain role during the execution of the exercise

  • Exercise Director, Moderator or Leadership Team: A person or team that directs the exercise. The responsibilities include setting up and dismantling the exercise environment, starting and ending the exercise, acting as the central point of contact for questions and problems which arise in the course of the exercise, facilitating tabletop exercises, managing the scenario, etc.

  • Monitor or Facilitator: The roles of monitors and facilitators are related and overlapping.

  • Observer: Observers are individuals or organizations who are invited to observe the exercise, without participating or monitoring performance.

  • Evaluator: Evaluators are the individuals involved in the process of evaluating the exercise.

After the implementation of the exercise, the organizer should evaluate the whole process based on the feedback of the participants. This feedback can include recommendations for a better planning and organization of future exercises. Based on the ENISA’s guide, this is the most important thing for the planning of the future exercise, as the outcomes of the evaluation are being taking into account by the organizers.

2.4 Overview – Comparison with ISO 22398

As stated in the ISO 22398 standard “This International Standard describes the procedures necessary for planning, implementing, managing, evaluating, reporting and improving exercises, and the testing designs to assess the readiness of an organization to perform the mission”. This standard gives the guidelines for organizing and implementing a successful exercise, [6].

The top management is responsible for organizing and implementing the exercise. The exercise should be planned based on its objectives. Also, the information and resources necessary to organize and conduct the exercise should be known in detail. In addition, the need analysis of the exercise should be included in the plan. The documentation of the exercise should describe how the exercise will be supported, how the organizer will recruit participants for the planning, and which are the necessary resources and budget. Also, any other element that can help to make clear and transparent to everyone the planning of the exercise could be added to the documentation.

Based on the standard, the size and the nature of the infrastructure set up the scope and the objectives of the exercise. Moreover, the secondary elements of the exercise, such as the complexity, the functionality and others, depend on the infrastructure type.

The standard states that there should be an exercise manager, a monitor, an evaluator and participants. Each of these roles has different responsibilities based on the tasks they need to perform. The exercise manager manages the exercise efficiently. The monitor and the evaluator measure the implementation of the exercise and the participants take part in the implementation.

At the end, the standard refers that there should be outcomes from the implementation, which the organizer will take into account for the better planning of a future exercise. These outcomes will be collected through the evaluation of the performance of the exercise project team, the ability of the exercise team to implement the exercise and the feedback of all the types of the participants, such as of top management, interested parties, exercise participants and exercise project team members.

Based on Table 1, it is obvious that the good practice guide of ENISA is compliant with the ISO 22398 in the area of the roles and in the area of the outcomes. The other methodologies should be reviewed and updated based on the needs and the requirements of the standard.

Table 1. Comparison based on the roles and the outcomes of each exercise methodology
Full size table

3 Process Comparison

The core element of an exercise is the conducting phase. The suitable and appropriate choice of staff, scenario and participants and the well-defined objectives of the exercise are necessary for a successful operational process. In this section, we present an analysis of the process which each methodology follows for the conducting phase.

3.1 The Conducting Phase of HERMES

The exercise using the Hermes methodology is separated into the following five phases: initialization, preliminary analysis, concept, realization, introduction and finalization [4]. In the initialization phase, a starting point is defined through a connection between the planning level and the operational level of the exercise. In the preliminary phase a decision is taken about the type of the exercise. In the concept phase, why this approach has been chosen should become clear and transparent and the exercise should be evaluated. In the realization phase, all the data and the documents for the implementation are prepared for the participants. In the introduction phase, the exercise is conducted, and evaluated and the report is drafted. In the finalisation phase, the organizer concludes the exercise in an orderly manner.

3.2 The Conducting Phase of MSB

In the MSB methodology, there are four phases for the operational level of the exercise. These phases are the plan phase, the do phase, the check phase and the act phase [7].

In the plan phase, the policies, goals, processes and routines are established. In the do phase, the policies, measures, processes and routines are implemented and enforced. In the check phase, the monitoring and auditing are performed by assessing, measuring and reporting. In the act phase, improvements to the exercise elements are realized.

In more detail, the check phase consists of eight stages. The outcomes of these stages are the input to the act phase. The stages are:

  1. 1.

    Appoint a head of evaluation.

  2. 2.

    Plan and organize the evaluation in collaboration with exercise management.

  3. 3.

    Formulate the evaluation questions and the basis for the subsequent analysis.

  4. 4.

    Train the evaluators.

  5. 5.

    Observe the exercise and conduct a direct feedback session.

  6. 6.

    Analyze collected material and compile evaluation report.

  7. 7.

    Present and disseminate evaluation findings.

  8. 8.

    Utilize lessons learned and begin planning the next exercise.

3.3 The Conducting Phase of ENISA’s Guide

In the good practice guide of ENISA, there is a distinction of the types of exercises into Discussion-based Exercises and Operations-based Exercises, [2]. This distinction is made based on the exercise tenor.

In Discussion-based exercises, the participants do not have any active participation. They only discuss the scenario and the procedures and test the decision-making. This type of exercises includes seminars, workshops, tabletop exercises, or games.

On the other hand, the Operations-based exercises test the procedures and check whether the staff is prepared to react and follow them. In this type of exercises, the participants are involved and active. Also, this type gives the opportunity for cooperation between the participants, who may be coming from different infrastructures.

The type of the exercise reflects on its size. If someone chooses the Discussion-based exercise, the size will not be as large as if the Operations-based exercise is chosen. If the size of the exercise is large, there exists the ability to test more procedures inside the infrastructure. The disadvantage of a large exercise is that the organizer should be ensured that the scenario covers all the objectives of the exercise, [2].

3.4 Comparison with ISO 22398

The ISO standard 22398 provides guidelines on the operational process of the exercise [6]. These guidelines are summarized below:

  • communicating the pertinent parts of the exercise program to interested parties, and informing them periodically for the progress,

  • coordinating and scheduling exercise projects and other activities relevant to the exercise program,

  • ensuring the selection of exercise project teams whose members have the necessary competence,

  • providing necessary resources to the exercise project teams,

  • conducting exercises in accordance with the exercise program and within the agreed upon time frame,

  • recording exercise activities and managing and maintaining documents, and,

  • completing after action reviews and following up on lessons learned and recommendations for improvement.

Based on these guidelines, this paper provides a comparison between the studied methodologies and the standard. As it is illustrated in Table 2, all the methodologies are compliant with the ISO standard.

Table 2. Comparison based on the operational process of each exercise methodology
Full size table

In addition, the standard provides guidelines about the type of the exercise. The organizer of the exercise should take into account the scope and the objectives of the exercise and then he should be able determine the exercise type [6]. The proposed types of exercise are summarized below:

  • Alert exercise: tests the organization by alerting the involved participants and by getting them to arrive at the designated place within a certain time.

  • Start exercise: tests how fast the emergency management organization can be activated and can start the carrying out of their tasks.

  • Staff exercise: increases the ability to work with internal processes, staff and information routines in order to create a common operational picture and suggests decisions.

  • Decision exercise: used to exercise the decision making process within an organization.

  • Management exercise: a combination of the alert exercise, the start exercise, the staff exercise, the decision exercise and the system exercise.

  • Cooperation exercise: coordinates the cooperation between the various management.

  • Crisis management exercise: simulates crisis conditions and tests the crisis management plan.

  • Strategic exercise: refers to comprehensive exercise activities at the strategic level.

In Table 3, a comparison between the studied methodologies and the standard are provided.

Table 3. Comparison based on the provided exercise types of each methodology
Full size table

Based on Table 3, one can see that none of the studied methodologies is compliant with the ISO 22398 in the operational process. Each methodology provides different types of exercises to be implemented. At this point, someone can raise an argument and could say that the methodologies may have included the non-covered types in the other types. In order for the methodologies to be compliant with the standard, each methodology should be reviewed and updated based on the guidelines of the standard.

4 Recommendations – Future Work

The industrial and critical infrastructures are organizations and facilities of major importance for the whole world. A failure on their functionality or an attack on these infrastructures is crucial and of high importance. The exercises are the mechanism with which someone can test the reaction and the procedures of an infrastructure.

In this paper, the currently used methodologies were presented and they were compared with the guidelines of the standard ISO 22398. It was observed that the studied methodologies should be reviewed and updated based on the needs, requirements and guidelines set by the standard.

Another point which was concluded as a result of this study is that the methodologies and the standard should set up restrictions and requirements as far as the participants are concerned. Only MSB requires that the participating infrastructure should implement and adopt a risk assessment and security policy, but even for MSB it is not obligatory [7]. The evaluated infrastructure should implement a risk assessment and adopt and implemented a security policy. Without this restriction, it is not clear how the procedures and the measures can be tested.