Keywords

1 Introduction

Space Situational Awareness (SSA) is a needed strategic and security capability, whose importance directly correlates with the increase in utilization of near/low-Earth orbit. Near/low-Earth orbit is accessible to over 70 space-faring nations, including even universities, businesses and startups [1]. Rapid advances in the use of near-Earth orbit by this increasingly diverse set of players, has led to that space becoming competitive, contested, and congested [2]. As these organizations make ever more extensive use of space, their space-based capabilities become part of the foundation of their economic strength where SSA is critical [3].

SSA is often seen as the ability to observe, track, and predict natural and inactive man-made objects in a defined space environment. Natural objects include meteoroids and near-Earth asteroids. Inactive man-made objects include spacecrafts, atmospheric re-entry vehicles, and debris. Since both natural and inactive man-made objects are nonresponsive to stimuli, they are collectively referred to as space debris.

Research has focused on SSA for space debris since several orbital collisions occurred within years of one another. In 2007, a KT-2 missile and a non-functional Chinese weather satellite collided. Meanwhile, in 2009 an active Iridium communication satellite accidentally collided with a defunct Russian satellite. These events resulted in debris which will stay in orbit for hundreds of years. Whereas the number of natural objects is essentially stable, the number of inactive man-made objects is growing exponentially. This increase has created a need for SSA to track space debris, as well as utilize the resulting information to avoid damaging effects to active man-made objects.

From consideration of such challenges, a more complete view of SSA emerges as the ability to create knowledge (i.e., understand and predict) the current and future state of natural and manmade objects in orbit around the Earth whose purpose is not the knowledge as an end in itself, but the application of such knowledge in managing controllable behavior to create desired outcomes [4].

Careful consideration of several aspects of this more complete view of SSA make it clear that the human SSA operator is the key to unlocking the puzzle of achieving SSA:

  1. 1.

    Not just integration of data into information, but instead transformation of information into knowledge

  2. 2.

    Understanding and prediction of space events

  3. 3.

    Managing behavior (i.e., command and control)

  4. 4.

    Achieving desired outcomes (where do desires come from?) [5].

The continued advances of sensor and network-based technologies will provide the human operator with unprecedented volumes of data, as well as meta-data regarding its sources, quality, and uncertainties. It is neither clear however to what extent human computer interface technology will enable the operator to make meaningful and valid use of the information, nor is it certain that any specific solution will be sufficiently resilient to function in the face of simultaneous exponential growth in [4]:

  1. 1.

    The number of objects in low/near Earth orbit

  2. 2.

    The number of government and non-government organizations interacting with said objects

  3. 3.

    The economic value of all of this activity, raising the stakes for all concerned [4].

If the only difference between unique decisions is the state of the world and systems provide their users information about the state of the world solely through sensors (certainly the case for SSA), a system that either replaces or supports cognitive processes of manipulating sensor data can provide game changing assistance.

Many SSA systems have been developed by commercial agencies and governments, both of which have their own limitations. For example, currently operators mostly base their assessments based on messages that are provided by government-owned radars and systems. Such systems have associated issues, such as: transparency, timeliness, and machine-machine interactivity within an assessment system [5]. To address said issues, operators have begun to utilize commercial space debris mapping services, which eliminate deleterious human-feedback loops by enabling the sending of requests and receiving of information services [5].

Although automation is the proposed solution by many researchers, the Architecture Driven Systems Laboratory (ADSL) research team strongly believes user interaction with machines is critical to solving the challenges posted by SSA because unlike humans, machines do not value the relationships between items in an environment. Machines, however, can detect any potential human errors. Although current space operators are able to constantly render updated images of the space environment and gather orbital data, the need for an integrated picture remains crucial [4].

2 Related Work

An SSA system of systems with dynamic composition exercising variable levels of cooperation requires an intuitive Human Computer Interface (HCI) to observe, orient, decide, and act on the large volumes of data produced by heterogeneous sensor networks; no one sensor can overcome the challenges of space object tracking. As such, SSA necessitates leveraging of Sensor Fusion techniques including. One effective method to realize the intersection of these fields is creating relationships of SSA network data using a dynamic ontology, which can define set processes to visualize the relevant virtual data in a Virtual Reality/Augmented Reality (VR/AR) environment.

2.1 Sensor Fusion

Collaborative architectures and services can alleviate the complications caused by dynamic space environment. Such complications include: motion of objects, rapidly advancing technological capabilities, and uncertainty in object location. Acknowledging uncertainty and limiting the user’s Situational Awareness (SA) so that it does not extend beyond the known situation is a critical aspect of the design of a system for sensor information fusion. Diminishing the risk of the user developing conjecture instead of awareness and exposing this disparity remains an active area of research.

2.2 Situational Awareness

The purpose of any SSA network is to create SA of some region of interest. Literature commonly refers to an SA framework consisting of those three levels, which was adopted in many domains including the space sector. This hierarchical structure propagates errors occurring in lower levels, yielding multiple errors in higher levels [6]:

  1. 4.

    Perceiving important data in the environment

  2. 5.

    Understanding the meaning of data and turning it in to information

  3. 6.

    Projecting the information to the near future [6].

2.3 Information Accessibility

Ontology is the study of the reality of an area; that is, it provides a definition of the objects and defines their inter-relationships. An ontological approach aims to formally represent knowledge components of the domain (also called a “knowledge model”) [7]. Creating an ontology might involve concept development, theoretical and philosophical development, computational development, as well as informatics, data management, and artificial intelligence techniques [7].

Machine Accessibility via Dynamic Ontologies.

Development of computational methods for ontology development helps researchers analyze and integrate data from different sources so that the fused sensor data is easier and faster to process. The literature related to collaborative SSA describes a lack of such integration in the field, which can and will be problematic for developing innovative SSA systems [7].

Despite a missing SSA industry standard dynamic ontology, some experimental ontologies are described in literature [7]. Capabilities afforded through the use of such ontologies included inference classifications for objects of interest, automated checks for relational consistency within the ontology, and renderings of the situation which aid in developing improved SA of a virtualized environment [7]. Therefore, to leverage these capabilities, SSA research should consider centering the design of an SSA system around the dynamic ontology to keep the system informed of the ever-changing situation in the overall scenario.

Human Accessibility via Immersion in Virtual Reality.

VR/AR is the proper approach for engaging the user’s sensory modalities as demonstrated by past work analyzing the hippocampal theta wave oscillations related to brain activity during navigation. Research using an EEG showed evidence for human movement-related theta low frequency cortical oscillations in immersive VR, and show that the most accurate measurements of navigation patterns come from real world immersion, then VR immersion, and then desktop VR [8].

Developing an SSA VR system centered on a dynamic ontology is a more “human” approach than one might think, as system users are already informally but constantly building ontologies in their heads about what is going on around them. They cannot help themselves – it is what people do to validate their encounters; psychology refers to such as a cognitive map. The more a user’s cognitive map is consistent with “truth” (the ontology that reflects the reality of the surrounding contents of the environment), the more the user can accurately understand the relationships encompassed within the environment. Users that insist on acting on their own cognitive map will create a different kind of Virtual Realty – one that is particularly unhelpful and likely to be harmful.

Rendering a VR environment based on all available sensor data is an immediate type of sensor fusion problem. However, the quantity and quality of sensor data is limited by the SSA sensor information infrastructure and SSA network architectures; the sensor data will never provide a full virtual view of the real world. As such, to develop an accurate, immersive situation, an SSA data presentation system must project information from both the provided data and inferred data. VR approaches can overlay the projected data over the virtualized environment data to enable distinction and fusion of information viewpoints.

3 Methodology

The Architecture Driven Systems Laboratory proposes the Space Situational Awareness Integrated Sensor Viewer (SSA-ISV): an extendable, interoperable rendering application created to enhance and explore a prototype dynamic analysis and simulation environment. This viewer connects to public and private SSA databases to help a user access relevant information using a tool which queries the virtual reality environment in-situ. This knowledge-enabled ontology-based simulation design environment leverages computational techniques to rapidly compare and employ to the SSA-ISV an interactive data presentation architecture based on virtualized, federated, faceted, registered, Resident Space Object point data (Fig. 1).

Fig. 1.
figure 1

Diagrammatic view of proposed approach

Full size image

3.1 The University of Arizona Architecture Driven Systems Laboratory

The Architecture Driven Systems Laboratory’s (ADSL) location at the University of Arizona enables collaboration with the SSA-Arizona program, which promotes international collaboration on a wide variety of SSA topics. Access to this expertise will allow for the development of a better-informed system if this project development is to continue; experts shall provide insights to enable space operators to tackle the numerous intricacies that arise in complex SSA.

The Systems Architecture Synthesis and Analysis Framework (SASAF).

The Systems Architecture Synthesis and Analysis Framework (SASAF) is the core product of the University of Arizona’s the Architecture Driven Systems Lab – but the real owner of SASAF is truly the interested technical community, which SASAF’s development methodology ensures. The SASAF expressly accommodates discipline specific tools and techniques per the tenants of Model Based Engineering and promotes architecture reuse across a multitude of projects by enabling specialization of generic object classes. SASAF Operator hardware is shown in Fig. 2.

Fig. 2.
figure 2

ADSL SASAF operator station hardware

Full size image

The SASAF core toolset allows for descriptions of countless possibilities of architectures by implementing software subdivided into three system life-cycle phases: the Architecture Creation Tool (ACT) and the Scenario Creation Tool (SCT) for the Design Phase, the Collaborative Command System (CCS), the Relevant Aspects of the World Emulator (RAW-E), the Integrated Sensor Viewer (ISV), and the Sensor Tasking Tool (STT) for the Operations Phase, and the Response Surface Analysis Tool (RSAT) for the Analysis Phase. For the purposes of Space Situational Awareness, the ADSL focuses on extending the ISV to develop the proposed SSA-ISV.

Figure 3 shows the functional flow block diagram for the three major portions of ISV and other parts of the SASAF and or the user community involved as much as they would like to be. Notice the upper branch feeding into ISV originates in the Air Force Research Lab (AFRL) and as such is opaque to those in the Space Situational Awareness (SSA) community who do not have immediate and in-depth access. It is not at all clear how AFRL generated the data files provided as part of the challenge. The motivation for incorporating this branch is to acknowledge that it is a necessary set of processing. Should AFRL decline to provide such data in the future, the SSA network instantiated by this study’s efforts would have to continue to acquire and post the data and then post processes it to arrive at the current AFRL input.

Fig. 3.
figure 3

The ISV functional flow block diagram

Full size image

Data Presentation Architecture Synthesis.

To create a dynamic analysis and simulation environment expressing data presentation architectures, the ADSL leverages architecture modeling techniques to represent data and its associated ontological semantic relationships within the ADSL ACT. The ACT defines the architecture at the start of system development through expression of many ontologies, the root of each being the different systems that are relevant to this study. One of these ontologies will always be the System of Interest (SOI), which will identify the sensors native to the SOI. But organic sensors may only be a small portion of the sensors the SOI can bring to bear. The other ontologies define the other systems in the study and must by their nature place their other systems in a range of relationship to the SOI.

Although the model structure required to frame a project may be well-understood and static, the parameters required for architecture execution, simulation, and analysis, may contain uncertainty, or be unknown altogether. These unknown characteristics can be fulfilled with the SCT, which shall identify remaining potential values that act as effectors or actuators to the environment. Iterating the chosen architecture through multiple scenario shall help the SSA-ISV operator develop awareness of how resilient each data presentation architecture is to a wide variety of conditions. With more scenarios, uncertainty in possible range of parameter values shall decrease. This decrease in uncertainty can be visualized in the SSA-ISV; users can react to this change by implementing dynamic alterations of object model behaviors, all while immersed within the virtual environment.

3.2 SSA-ISV VR Operations

This research aims to transform the ADSL capability to define and utilize presentation architectures for the provided AFRL data. Within systems engineering, this research is a key enabler for Model-Based Systems Engineering (MBSE) which is the international systems engineering community’s objective, and beyond that to Model-Based Business, that is integrating technical, management fields for a revolution in how systems are developed and deployed. The SASAF aims to reduce the time and effort required to find suitable arrangements of hardware, software, procedures, and policies for a wide range of domains. This will enable addressing problems on a timely basis utilizing limited resources. To achieve this goal, this research has begun creation of an extensive ontology and prototype to properly handle industry provided object ephemeris data.

Dynamic Ontology.

To generate contextually accurate results within the SSA-ISV, one must make sure to incorporate all relevant aspects of the simulation for tools. The SSA-ISV makes use of databases and other information technology but requires that every piece of information be available in a flat file, thereby reducing the information technology expertise demands on small players in the SSA industry. The ISV, which is inherently extendable, moves beyond the desktop to take full advantage of augmented and virtual reality interfaces. To enable users to utilize these capabilities, the ADSL develops a method by which to identify space objects within the provided AFRL dataset.

The Systems Tool Kit (STK) software by AGI Inc. contains a standard space object catalog. Most of the elements from this catalog are made public, but some are restricted. AGI provides the publicly available information for use with their STK software in the form of satellite database files and two-line mean element files (TLEs). The AFRL dataset also contain publicly available TLE elements. As such, if the ADSL can register each data object to real space object data found to more easily extract sensor and instrument data as needed.

Sensor Fusion.

The commercial company LeoLabs was created to secure commercial operations in Low Earth Orbit (LEO). As the LEO ecosystem around the Earth is getting more crowded, the risk of collisions rises. To monitor this, LeoLabs installed a worldwide network of ground-based, phased-array radars that enable high resolution data on LEO objects. The LeoLabs platform delivers applications for operating in low Earth orbit; data products can be accessed through variety of interfaces such as web based, command line and REST API.

In the completed system, a user can specify which information they are seeking using filters related to common object parameters such as location, speed, size, etc. Furthermore, the user can even alter the virtual reality to view the effects that an actuation such as satellite repositioning would have throughout the multiple levels of Situational Awareness. The SSA-ISV shall then suggest views based on ontology recommendations formulated from user preferences and actual and projected data. Selecting a point of a view sends a signal to the simulation, actuating a response to the overall environment surface at that time, then simultaneously localizes and maps the frame of reference to the stakeholder’s intended viewpoint.

Figure 4 shows a 3D virtual model of the Cassini satellite mission within the prototype environment. The ADSL posits that having an interactive virtual environment will increase the understanding this situation, and better allow for collaboration transdisciplinary collaboration compared to traditional document-based reporting.

Fig. 4.
figure 4

Sample 3D model in SSA-ISV prototype

Full size image

4 Results and Discussion

The SSA-ISV prototype correlates object location, position, and velocity data from the provided CSV files to corresponding NORAD IDs within each object file name. The SSA-ISV compares the NORAD ID to the LeoLabs database based on user requests and returns both the LeoLabs object ID, as well as the object name to the virtual environment. An extended version of the prototype can allow for comparison of the object names to other science APIs and use similar methodologies to help both humans and machines develop SA of the data.

4.1 Situational Awareness Affordances

The ADSL has successfully created a Virtual Reality environment which allows a user to interact with the provided dataset. The enhancement of SSA using dynamic ontologies and virtual reality environments through procedures developed in the University of Arizona’s Architecture Driven Systems Laboratory provides a means to fuse, observe, and augment the activities of AFRL/RY COMPASE Center’s various services, including: Modeling and Simulation, the Sensor Data Management System, Test & Evaluation, and the Virtual Distributed Laboratory [9]. In this way, the ADSL’s methodology enables both machine and human understanding of AFRL’s data.

The prototype currently simulates a virtual reality environment using the software Unity comprising of three satellites in orbit using the provided AFRL dataset. The user can move through the space environment to position himself/herself near objects by facing a direction and touching the top portion of the HTC Vive right controller’s trackpad. Clicking the trackpad moves the user faster through space. Selecting an object acquires information from LeoLabs using an API key. The returned JSON data is parsed into objects using regular expression, resulting in the ability for the user to see the NORAD ID, satellite name, and LeoLabs ID in VR. The user can select the object by pressing the grips on the right controller to start the selection and pulling the trigger to complete it. Information of the selected satellite is presented on a card above the left controller.

4.2 Sensor Fusion Affordances

If the SSA system design centers on ontologies that come into being through computational techniques, then one immediate advantage of this is that every part of the system can access and exploit the ontology; this is the key idea as to how the ACT communicates to the rest of the SASAF. Some systems will assert in their architecture that they are able and willing to share sensor data with the SOI; such systems amount in extensions of the SOI. Other systems will have no method by which to share sensor data, and still other systems may be ready and willing to reduce the effectiveness of the SOI’s sensors through techniques such as radio and radar jamming across different bandwidths.

Depending on the driving objective of the study, the architecture definition may become quite detailed, especially because heterogeneous sensor networks have different procedures by which to exchange data. Once the architecture defines these procedures, data becomes accessible by every other component of the system as appropriate. At this point, the ADSL Sensor Tasking Tool (STT) enables the user to command only the sensors the architecture permit to better accomplish desired tasks and develop deeper SSA. Likewise, the SSA-ISV does not have to determine what sensors are available to make the fused sensor picture, because the architecture has been predetermined in the SASAF design phase.

4.3 Prototype Information

To ensure that the prototype meets the goals described in this research, the ADSL has created an ISV Specification document to keep track of requirements that shall be satisfied to accomplish the goals of the SSA-ISV. Furthermore, the ISV Verification document provides a scheme by which to verify that the proposed requirements are satisfied. Documentation for implementation are included within the Implementation document, which includes some steps taken towards verification of the Implementation with the requirements in the Specification. The SSA-ISV prototype executable is compatible with the HTC Vive and can also run on non-VR desktops. The prototype design allows the relevant files to be shared across research computers independent of the sizeable AFRL dataset.

5 Future Work and Conclusion

Currently the SSA-ISV prototype has demonstrated the potential to grow into a powerful tool for SSA purposes. Future work must be completed to enhance the working prototype, remove glitches, and add capabilities. Scaling the prototype using machine automation techniques will eventually allow for the full population of a novel, innovative, knowledge-based environment. Real-time capabilities shall be included as well, interfacing with sensor network information available throughout the scientific community. The ADSL’s approach innovation involves gathering data from different sources and integrating them using Sensor Fusion technique into a holistic picture incorporated into a VR modality. This solution goes beyond visualizing the data: it provides an entirely virtual environment, within which the user develops a thorough SA.

The ADSL has generated the ISV Development Plan document to improve organizational transfer of the SSA-ISV prototype and better satisfy the needs of stakehodlers. This development plan also includes future activities to be performed to incrementally enhance the SSA-ISV prototype. Such plans include enabling the user to project the most likely complete view of all objects in cases where the available sensors are insufficient to create a full 3D view of an object in the virtual environment. Other future development plans include integration of the ADSL Sensor Tasking Tool to give the user control over any sensor available to the system of interest.

Virtual environments, based on dynamic ontologies which define relationships between system parameter evolutions over time, can act as a replacement for the natural environment. Using sensor data, a machine can construct a stable yet dynamic virtual environment via tasks such as projecting future states, evaluating courses of action, considering improbable outcomes, and so forth. As such, creating an immersive environment to analyze space system data enables space operators to develop a deep SSA through the ADSL’s proposed SSA Integrated Sensor Viewer (SSA-ISV).