Keywords

1 Introduction

In recent years, the human factors of cyber operations and cyber security has gained increased attention [1,2,3,4,5]. The acknowledgment of the human as ‘the strongest link’ is more common [4, 6], due to the notion that humans remain superior to technology when it comes to engaging in macrocognitive work (e.g. adapting to complexity through problem identification and sense making in ambiguous, shifting conditions) [7]. Humans now operate extensively in hybrid contexts, characterised by cyber and physical reciprocal determinants [8], merged with tactical and strategic level interaction [9]. More human computer interaction is placing higher demands on humans to establish cross-domain situational awareness [10] and to govern technology, whilst simultaneously complying with physical environment demands. Task demands in hybrid contexts exceed what we used to consider ‘enough’ to cope in a digitized context, and performance measures need to include soft skill proficiencies that can traverse digital and physical domains [11]. Research and understanding of the cognitive processes that support mastery of hybrid contexts are still scarce [5].

To learn more about individual cognitive manoeuvring requirements, the researchers monitored cognitive dynamics of cyber cadets in hybrid context. This paper first explains how we operationalized cognitive focus (i.e. cognitive location) by utilizing The Hybrid Space conceptual framework [9]. Then development of a self-report software, The Hybrid Space app, to help capture, visualize and analyse the cognitive focus of individuals and teams operating in hybrid contexts is presented. Further, an example describes the context in which the software was applied to capture cognitive focus of a cohort of cyber cadets at the Norwegian Defence Cyber Academy (NDCA) participating in a four-day Cyber Defence Exercise (CDX). Examples of collected data are presented and the applicability of the software is discussed.

2 Capturing Cognitive Focus

Cognitive focus can be understood as an aspect of attention that involves bringing selected information into conscious awareness [12]. Cognitive agility can be understood as the ability to be attentionally flexible, where flexible expansion and contraction of cognitive focus allows for both panoramic and selected attention in The Hybrid Space [13]. To be able to scientifically address the cognitive agility levels of personnel operating in hybrid contexts, cognitive focus has to be measured first [14]. Capturing the cognitive focus and concurrent thinking processes of individual cyber operators is a challenging task, presenting a variety of factors that can distort accuracy and validity of data. A number of methods designed to collect such data, aiming to get insight into thinking processes exists.

Behavioural task analysis is not applicable to capture cognitive focus, as this kind of information is not directly observable [15]. As a consequence, indirect techniques like visual search pattern have been utilized for tapping into the nature of expert cognitive processing in physical environments [16]. In complex domains though, experts also rely on sense making processes that couple chunks of information emerging from the cognitive domain [17]. This lead to Cognitive Task Analysis (CTA) and Cognitive Work Analysis (CWA) being developed to access cognitive elements of experts in action [18, 19]. Both are known methods to capture the cognitive processes in cyber operators [20, 21], but involve either in the moment eliciting techniques (e.g. speaking out loud while performing tasks), or retrospective methods of data collection like ‘the knowledge audit’ or ‘the simulation interview’, aimed at uncovering cognitive demands or skills required for expert task performance [22]. While in the moment eliciting methods might be intrusive and reduce operator performance, retrospective methods might present difficulties in recalling specific cognitive focus over an extended period of time. Despite intrusiveness, CTA are proven methods for eliciting the cognitive task requirements and capturing the covert cognitive processes experts use to perform complex skills [18].

Non-intrusive techniques like functional magnetic resonance imaging (fMRI), electroencephalography (EEG) and eye-tracking are also used to assess cyber operator performance [23]. While these methods are less intrusive and can give valuable data of cognitive load and aspects of cognitive processing [24], neither give access to the kinds on cognitive focus and specific sense making processes the operator needs to accomplish in hybrid contexts.

These examples of performance data collection methods give access to important task-knowledge, cognitive load, neurological processing and what information cyber operators seek. But they do not elicit knowledge concerning how cyber operators cognitively focus and cognitively manoeuvre over time in order to make sense of the continually evolving hybrid context. Neither does it give knowledge of what skills are used to regulate cognition, nor what cognitive dynamics are beneficial to support performance in hybrid contexts. Further, in complex domains the construct of performance itself is questioned due to the complexity of interactions [2, 14, 16, 25, 26]. So far, measuring performance in cyber exercises has relied on performance measures like ‘capture the flag’ or other types of hits, errors, accuracy and time (HEAT) measurements or subject matter expert evaluation [27,28,29].

Embracing the full complexity of human computer interaction in hybrid contexts reveals the need to explore and measure emergent properties [30]. We argue that performance in hybrid contexts may be dependent on indicators and predictors like cognitive agility levels [14]. For now, capturing the immediate cognitive focus of personnel engaged in macrocognitive work in hybrid contexts requires self-report. In this paper, we introduce a software developed to measure cognitive focus and analyse cognitive agility.

3 The Hybrid Space Conceptual Framework as a Tool for Measuring Cognitive Focus

The Hybrid Space conceptual framework (see Fig. 1) describes the influencing factors on individual psychological conditions in hybrid contexts [9]. Hybrid refers to the complexity of interactions between agents in the cyber domain and the physical domain in this space, at all levels of hierarchy. The Hybrid Space draws attention to the human as the converging point of sense making, acknowledging the reciprocal deterministic relationship between environment, cognition and behavior, whilst acknowledging individual agency to self-influence cognitive activity [31].

Fig. 1.
figure 1

The hybrid space conceptual framework [9]

Full size image

The x-axis visualizes the reciprocal relationship between the physical domain on the right-hand side, and the cyber domain on the left-hand side. This subjective assessment can be seen as the cyber operator’s current cognitive focus in relation to the cyber and physical domains, and can be measured by self-reporting the location of the momentary cognitive focus. Simultaneously sense making requirements between tactical and strategic considerations the can be marked with y-axis position. For example. when analysing malware, tactical and local considerations might be applied to sense making processes, while considering attribution might require a strategic (e.g. national or geopolitical) cognitive focus [32].

To allow for measurement of cognitive focus using The Hybrid Space app, the conceptual framework is mapped in a Cartesian plane. The x and y axis movements are limited to −100 to +100. Based on the cognitive focus of each movement, The Hybrid Space is operationalized through four indices (See Table 1, and Fig. 2). These indices are quantitative proxies for the construct of cognitive agility, and can be utilized as dependent variables in the case of further statistical analysis [14]. When developing The Hybrid Space app, two sliders and a text window were added to allow for collection of additional data of interest. The sliders are comprised of a 9-point Likert scale (See. Fig. 3), and the slider text can easily be changed at the researchers’ prerogative.

Table 1. Explanation of indices
Full size table
Fig. 2.
figure 2

Operationalization of the hybrid space movements [14]

Full size image
Fig. 3.
figure 3

The hybrid space app user interface

Full size image

4 Software Description – The Hybrid Space App

To develop the software, a spiral lifecycle methodology was used. This allows for continuous feedback on the development process. This involved creating an initial stripped-down prototype of the software based on the research requirements. The initial prototype was then tested to identifying strengths, weaknesses and risks with the current prototype. Changes were then proposed for the next revision of the prototype, and a new revision of the prototype created. This process ensured that a final product could be created in collaboration between researchers and programmer.

The software is based on three main components; a backend database component, a database interaction component and a front-end component. The backend database is where all the collected data is stored, in addition to also being where all the business logic of the system is processed. This component is based on Postgres (PostgreSQL), an open source object-relational database management system. Postgres handles all equations used and calculations done by The Hybrid Space app through a set of triggers that are activated when new data is added to the database. The database structure and triggers are defined in a simple database initialization script that Postgres processes upon initialization of the database. This file can be located at /Docker/hybridspace_db/initdb.sql in the source code.

The backend interaction component allows for user interaction with the database. This component is built using a web application framework known as ExpressJS. The component handles all queries to the database and is a RESTful service, listening for requests over HTTP and replying with JSON data. Creating a RESTful service standardizes the way users and other applications interact with the system. Some examples are shown in Table 2.

Table 2. Hybrid Space REST API examples:
Full size table

This is currently the only way to add, modify or delete data in the database. The query possibilities can be found in the api routes definition file found in /routes/api.js. The functions and their associated database queries can be found in the file queries.js , located in the project root directory.

The last component of the system is the front end. This component gives the user an interface to make REST queries to the server, and parse the JSON data the server replies with. The current front end is web based, using JavaScript to send and parse data. In the future, this component could be based on a native Android or iOS application due to the flexibility associated with using a REST API for server communication.

When new data is added to the system, a set of calculations are done. The distance traveled (HSDT) is calculated using the movement along the x-axis (HSxM) and y-axis (HSyM). HSxM and HSyM are calculated by finding the absolute value between the new plot x/y value and the existing x/y plot found in the database. The HSDT value is then calculated using basic trigonomics:

$$ {\text{HSDT}} = \mathop \sum \limits_{i = 1}^{n} \sqrt {({\text{HS}}x{\text{M}}i)^{2} + ({\text{HS}}y{\text{M}}i)^{2} } $$

This calculation can be found in the function ‘trg_travel’ function located in the initdb.sql file. A snippet of the code performing the logic can be found in Table 3.

Table 3. Calculation of distance
Full size table

The system also keeps track of what quadrant the user resides in so that any quadrant changes can be registered. This calculation is done using the function ‘trg_quadchange’ located in the initdb.sql file. A snippet of the code performing the logic can be seen in Table 4.

Table 4. Calculation of quadrant change
Full size table

This data allows for more advanced queries to the system, such as how many quadrant changes a team has, or how much travel a specific person has.

4.1 Installation

The software was developed to run in a Docker container, however it is also possible to install the software directly on a Linux host without any containerization. Currently, the software has only been tested on Fedora 24/25/26, however other operating systems that support Docker should function just fine. Docker is supported on most Linux distributions, with limited support for Windows hosts. It is therefore recommended to run the software on a Linux distribution such as Fedora, CentOS or Ubuntu.

As mentioned earlier, it is possible to run the system directly on a Linux host without any containerization, however the authors recommend using Docker as it simplifies the installation process significantly. The Hybrid Space wiki documentation included with the source code details this process. In short, this process involves installing nodejs and postgresql on a server, setting up the database access and running the application.

The Hybrid Space app is freely available for download at http://github.com/metrafonic/TheHybridSpace. Guides to installing and using can be found in the connected Wiki.

5 Collection of Cognitive Focus During a Cyber Defence Exercise

Collection of cognitive focus was performed during the annual CDX at the NDCA, November 2017. A complete cohort undergoing a cyber engineer education totalling 38 cyber cadets participated in the CDX. They worked in four independent student teams (one team consisted of 9 or 10 members) in four separate rooms. 23 cadets participated in this research using The Hybrid Space app.

All of the participants were subject to the same external cyber and physical activity, framed in the same scenario. Ongoing momentary assessment of perceived cognitive focus was conducted as cadets were instructed to mark their cognitive location in The Hybrid Space around every full hour (0800–1800). Simultaneously the perceived level of control and cognitive effort were assessed by adjusting the sliders (see Fig. 3). Comments were made voluntarily in order to minimize intervention time. The assessments were repeated for four consecutive days throughout the course of the CDX, giving a total of 854 data entries in The Hybrid Space app.

The CDX is constructed to challenge participants to apply cognitive agility to make sense of the multi-domain reciprocal dependencies in The Hybrid Space. This is achieved by a using a scenario where both strategic situation information and tactical information is injected as part of a holistic scenario driven events matrix. In addition, cadets are challenged to communicate their current multi-domain situational awareness in relation to the overall evolving scenario.

6 Examples of Research Results with the Hybrid Space App

During the course of the four day CDX researchers were able to monitor the entry of data to ensure adherence to instructions and reduce the possibility of missing values. The Hybrid Space app graphical output is shown in Fig. 4 with the complete movement of one person during the four days, as well as variations in control and effort easily accessible and visualized. In a real-time analysis, this gives access to individual cognitive focus, where participants struggle, have control or where they put in effort. Compared with the Exercise Control (EXCON) knowledge of scenario developments and task requirements this data can shed light on how participants manoeuvre and focus to make sense of information emerging from cyber and physical domains.

Fig. 4.
figure 4

Example of data collected with The Hybrid Space app

Full size image

Each data entry comprises of an automatically generated ID number, username, team name and time stamp. User entries comprised of x location, y location, control, effort and voluntary comment. Raw data backend is shown in Fig. 5.

Fig. 5.
figure 5

Data entries administrator view

Full size image

An additional view, Fig. 6, showing the aggregated number of data entries of each person and each team was devised to give an easier overview of the data collection process, as well as the possibility to export data on individuals and/or team. This reduces data handling from raw data as well as provides an overview that can be useful when comparing the data on team or group level.

Fig. 6.
figure 6

Data entries team and group administrator view

Full size image

In contrast to paper-pencil solutions that we have used in early stages of cognitive agility research, The Hybrid Space app proved to be less invasive and removed manual data transfer errors. During the period of data collection, preliminary data analysis can be conducted in real-time, and the cognitive dynamics can be visualized instantly.

7 Applicable Contexts and Further Development

The application of The Hybrid Space app in combination with The Hybrid Space conceptual framework in the context of education and training of cyber cadets can give insight into so far unexplored cognitive dynamics on individual and group level. From an EXCON perspective, the data might be useful for observing cognitive focus during the course of a CDX. The software can also be used for debriefing and/or as a complimentary tool for conducting CTA after a training session is completed. Further research can be conducted applying statistical analysis of The Hybrid Space movements with data from other inventories measuring i.e. self-regulation, metacognitive awareness or other constructs known to support cross domain performance [13]. Such combination of data can shed light on beneficial cognitive traits and competencies supporting agile manoeuver in The Hybrid Space [13, 14]. Over time, and in combination with valid performance measures, further research can produce knowledge about cognitive skills that are beneficial for operating in hybrid contexts.

The visual representation given in the administrator view (Fig. 4) gives an overview of the predominant cognitive focus at an individual level (i.e. direct access to a part of the cognitive dynamics of each participant). With further development, this could also be expanded to visualize predominant cognitive focus at various group levels (e.g. team and unit). As cyber operator performance is dependent on both individual (metacognitive) and team (macrocognitive) work [9, 11, 19], collection of such data could contribute to develop knowledge and understanding of beneficial cognitive focus and dynamics displayed by cyber operators in hybrid contexts. However, using The Hybrid Space for research purposes demands a level of prior understanding among researcher and research objects. Pathways for improved understanding of hybrid contexts among cyber cadets is a developing area of research [11, 13].

While The Hybrid Space app is developed utilizing the military cyber operator practice, there are opportunities to adopt the framework, and the software, to research other digitally mediated cyber-physical contexts where humans operate in and through digital technology. An example could be to research how digitally mediated learning platforms connected to cyberspace influence the classroom context and impacts teacher-student power gradient and dynamics. Or to research the cognitive dynamics displayed by parents and children in the family practice as they are influenced by more time spent in the world of cyber.

8 Conclusion

The Hybrid Space app is a software tool providing the researcher with a developed software and method of capturing and visualizing momentary cognitive focus and the dynamics of individuals in hybrid contexts. Compared to other methods of cognitive data collection like CTA, fMRI or EEG, using The Hybrid Space app gives access to new and qualitatively different data on individual cognitive dynamics with a minimum of intrusion. The software further provides the opportunity to visualize cognitive dynamics for research, teaching and presentation analysis. The Hybrid Space app provides necessary computation options of variables and displays various measures of movement in The Hybrid Space, on individual and group level. It also provides access to easy research setup and use, due to its graphical interface and versatile platform possibilities. Finally, researchers are provided with access to export all collected data with timestamps in comma-separated values (CSV) format. The Hybrid Space software provides a graphical user interface that makes it applicable for both research and teaching purposes. An open access approach and compatibility allows further development and/or integration with other software tools maximizing benefits by having possibility of combining various software solutions.