Keywords

1 Introduction

Data visualization, their analysis and interpretation, are areas of high interest to the scientific community [1, 2]. In our Big Data era, massive amounts of data are processed all the time, some of them leading to scientific discoveries [3]. Therefore, visualization, in order to detect patterns and relationships, has become a research avenue of great importance.

Spatio-temporal data are the main data sets that are analysed; they are related to the evolution of information in two key variables, space and time. By analysing these data, can be known where and when they fluctuate. There are many applications and studies [4, 5] related to the visualization of this type of data, which require special strategies for the graphical representation of information with multiple dimensions.

In the 2014 Ontology Summit [6] it was concluded that the use of ontologies to represent information models and infer knowledge has played a key role in the development of new semantic technologies. However, the collaboration between the Semantic Web and Application of Technologies communities has not been as active as one might expect. In this Summit, the existing problems and means to solve them were defined. Due to the application of these solutions and the need to understand the meaning of the data, in order to process them in areas such as Big Data, the problem is being solved [7, 8] and there is an increasing number of projects in this area that use ontologies [9, 10].

The term ontology is defined for the first time as: “An ontology defines the basic terms and relationships that comprise the vocabulary of a thematic area, as well as the rules for combining terms and relationships to define the extensions of the vocabulary” [11].

Most data visualization software tools are standalone applications, which handle small data sets [12]. In addition, they do not usually use data that are in an ontology, neither to show them nor to know how to do it.

Traditionally, speaking about graphic visualization of ontologies has been synonymous with visualization of graphs, tree maps and similar techniques that have developed a multitude of standalone and online tools. These tools are used to see the ontology and navigate through it to see its instances [13]. Recent projects use ontologies in order to know how to show the content of an ontology [14, 15]. One of the most advanced is the one proposed by the VISO ontology [16]. This ontology, together with the RVL [17], defines how to visualize the information and interact with it in a user-definable graphical environment.

In this paper a wide related work section is depicted, where are described the advances in the research lines involved in the design of SILKNOWViz. Next, in the motivation section, the limitations in these research lines are listed, and ways to alleviate these problems are introduced. SILKNOWViz, a tool developed within the SILKNOW project, is proposed as a way to achieve this objective. In the section on design and methodology, some considerations on the design of the tool are described, as well as the graphic interface and the first developments. Finally, the conclusions of this work are exposed.

2 Related Work

The visualization of information helps the process of obtaining knowledge and a better understanding of the information through the graphical representation of massive data sets [4]. In the case of spatio-temporal models, the represented data fluctuate in space and time. They are data sets that are difficult to visualize, especially the more dimensions there are, in addition to space and time [5]. In order to achieve this objective, the data have been structured in different levels in the last two decades. A series of operations are defined for each level to be performed. These classifications define what can be queried to the system and what can be obtained in each query. These studies have been carried out in conjunction with the evolution of the tools and user interface elements that allow consulting data, as well as viewing them. Next, is depicted an analysis of some of the most widespread classification techniques, as well as the different types of developments.

2.1 Classification of Spatio-Temporal Data

Peuquet talks about the existence of three main parts in spatio-temporal data: where, when and what [18]. These parts describe a location, a time and the data to be represented. On the same type of task-oriented model, more extensions were added; for example, Andrienko [19] classified the tasks to analyze the information consulting by points, regions and trajectories. With the evolution of technology the data has changed, the size of the information is much larger and they have many more attributes to consider. It is important to consider objects and their attributes. Nowadays, with massive amounts of available data, more attributes can be considered, helping with the identification of patterns.

For this reason, new ways of classifying data arise, such as that proposed by Guo [12], which adds an additional part to the three proposed by Peuquet, the “who”, which refers to objects, or series of objects. In addition, Guo associates the “what” part with the attributes of these objects. The four parts can be combined around four levels, performing queries with more to fewer variables, so that the fewer variables used, the more information will be shown, reducing the certainty of the analysis. A table with the different tasks for each level can be seen in Fig. 1. These new ways of classifying data generate new results that allow better analysis of the data and discover relationships, or patterns, in an increasingly simple and rapid manner. However, since reality is very complex, new visualization demands appear as the system improves.

Fig. 1.
figure 1

Source: Guo [12]

List with the tasks contained in each task level.

Full size image

A new requirement appears when objects are both objects and attributes. This happens when an object is related to other objects. In this case, the related objects are new objects and attributes of the source object. An example would be manufacturers and manufactured objects that evolve throughout space and time. It is very interesting to see where these objects have been manufactured, but also where they have been used and what they are related to. If we want to see the evolution of this evolution in space and/or time, the problem is even greater and requires further improvement of techniques.

2.2 Visualization Tools

Since the beginning of this research, many software tools have been developed to visualize information [1, 2, 4]. In the last two decades, due to technological advances, the use of three-dimensional representations with a high level of interactivity has increased.

Some developments show information (what + where + when) over a bi-dimensional or three-dimensional region. These tools use various techniques such as colour maps, colour temperature patterns and clustering techniques [22, 23]. Colour, cluster size, the number of dimensions (2D/3D), elevation, and even the use of additional labels, are the visual elements that are used to represent information in the graph (see Fig. 2).

Fig. 2.
figure 2

At the top, screen shots of a web application that uses coloured regions and colour temperature maps, to visualize the incidence of cancer mortality in Taiwan. Source: [20]. Below, screenshots from tools that use clustering techniques to represent the information. Source [21].

Full size image

In addition to visualizing information and displaying common form controls in the user interface, innovative controls have also been created. These controls are used in order to define and filter the time in which the information to be displayed is represented. One of the most used interactive devices is the timeline [24, 25], which contains information on the different time levels. It starts from a broad level, usually in the top part, to go down interactively, until selecting a concrete period, in the lower zone. It is often used in cultural heritage visualization tools [26]. See Fig. 3 for examples of timelines, specifically, a Time Wheel control [19], which allows the selection of specific months, as well as days and a time interval in a simple way.

Fig. 3.
figure 3

Left and right show screenshots of spatio-temporal data visualization applications in cultural heritage application. Source: [24, 26]. Screenshot of a Time Wheel control in the upper central part. Source: [19]

Full size image

Analysing how information evolves over time can be the main goal in determinate information systems. In this case, it is important to see the date in different time intervals. To solve this problem, the scientific community has also developed several solutions. Some are based on three-dimensional spaces, like space-time cubes [5], or spirals [27], which displays a 3D spiral on the objet, where each lap represents a period of time. Spiral shows a lot information, but it is very difficult to analyze it. Other are based on two-dimensional spaces, like flow maps [28], used to represent trajectories, or movement of objects in time. Finally, animations are also used, where each frame corresponds to a specific period. See some applications that use these techniques in Fig. 4.

Fig. 4.
figure 4

The upper part of the image shows the different frames, one per year, of the incidence of terminal cancer in Taiwan, resulting in an animation. Source: Adapted from [20]. In the lower left zone a space-time cube is displayed, where a horizontal plane (map) can be moved. Source: [19] To the right, two screenshots of an application that use 3D spirals, where each lap is a period of time and has additional information related with the appropriate year. Source: [27].

Full size image

The advance in the use of these techniques has been important, especially in the graphic quality and the possibilities of interaction, because many of these ideas have been developed several decades ago. One of the main problems in this area is that developments are often linked to a particular application. For this reason, it is difficult to reuse the tool and frequently the only reused components are the web services used to show maps, or satellite images, such as Google Maps and others [29].

2.3 Application of Ontologies in Visualization Systems

Typically, Big Data tools process information without need of a high semantic content. For instance, there are image-processing tools, where the content of images is analysed using neural networks, looking for patterns in the image, and taking as a reference a large sample of correctly classified images [30]. Probably, if semantic information was available, the process would be faster, because this information could contribute to avoid higher computational cost processes.

Ontologies are used to represent complex models of information and to infer knowledge from them, because a high semantic content can be defined in the representation. In 2014 Ontology Summit [6] it was stated that although ontologies have played a key role in Semantic Web technologies, their application in new lines, such as Big Data, is lower than expected. It seems logical that both lines of research should have had a greater relationship, but this has not happened, in fact. Seemingly, the measures proposed in the 2014 Ontology Summit to increase its application in this type of projects are beginning to yield results in recent years [8]. New projects that combine the results of these two lines are in operation [7, 9, 10] and the tools that allow access to the data of an ontology are also evolving [31, 32]. If work along these two lines follows the expected path, the information to be visualized, or at least an important part of it, would be in an ontology. Therefore, the visualization of spatial and temporal data in this type of projects could take advantage of the results obtained in the graphical visualization of ontologies. However, the main activity of this field consists in the visualization of the structure and content of the ontology by means of interactive representations of graphs, treemaps and similar techniques [13, 33]. In these works, the main objective is usually not the analysis of content in order to discover patterns, or to deduce complex relations. For this reason, these investigations may provide components that are useful in the study of spatio-temporal data, but they have not been developed as much as the techniques used in this line of investigation.

In the last decade, new research proposes the use of an ontology to specify how its content should be visualized, as well as for defining the user interaction [14,15,16]. One of the most generic and reusable results is the ontology Visualization Ontology (VISO) [16]. This ontology allows the definition of its concepts considering how to visualize them and how to interact with them. This ontology was developed as the basis of the RDFS/OWL Visualization Language (RVL) [17].

The problem with the VISO ontology is that it involves close consideration of visualization aspects in a two-dimensional environment, but hardly delves into 3D visualization components. It would be necessary to create new classes and extend others to consider three-dimensional representation, which is fundamental in many spatio-temporal data visualization techniques.

3 Motivation

After reviewing previous publications, the following issues and conclusions arise regarding the visualization and analysis of spatio-temporal data:

  • More work needs to be done on the relationships between different objects, objects that are at the same time objects and attributes.

  • It is necessary to create more open and multiplatform tools that can be used to represent and analyse spatio-temporal data. These tools must be capable to be embedded in other projects, especially in web applications.

  • The use of ontologies to represent the information to be visualized should be encouraged. In addition to values, data can contain semantic information, allowing new possibilities of interaction and visualization.

  • Generic tools have been created in order to define how the information is to be displayed. VISO ontology, one of the most advanced, needs to be extended to contemplate more deeply the needs of graphics in three-dimensional environments.

In order to satisfy these needs, we propose to carry out the following tasks:

  • To eliminate limitations in the three-dimensional environment of the VISO ontology and the RVL language, as well as including activities and interactions in order to implement techniques for the representation of spatial and temporal data.

  • To define a way to visualize relationships between objects, which allows visualizing the evolution of these relationships in different moments of time, simultaneously.

  • If the data are represented by an ontology, use the VISO ontology and the RVL language to define how to visualize the information.

  • To develop an open source software tool, in a multi-platform development system that allows to visualize the results of any project, especially on web platforms.

These tasks will be developed within the SILKNOW project, by designing and developing the SILKNOWViz tool.

The SILKNOW project is an interdisciplinary project, funded by the H2020 Programme of the European Union, in order to preserve and promote the heritage of silk textiles. SILKNOW will take digital silk textiles’ data from the databases of several institutions, from online catalogues and from APIs (where available). These data will be analysed and processed with advanced text analytics and image-based deep learning techniques in order to homogenise their content, automatically retrieve semantic information, complete poorly tagged data, and translate the text into four languages.

The CIDOC Conceptual Reference Model [34] is used in SILKNOW. The selected implementation is the Erlangen CRM/OWL [35], a RDFS/OWL ontology.

One of the results of SILKNOW is the implementation of a public web site where the data of the project can be searched. In this website, the queries’ results can be visualized through a 2D/3D component with an interactive map. This component is the SILKNOWViz tool. Next, we describe how to manage relationships between different, related objects in a spatio-temporal data model. Taking into account these considerations, the design of the tool user interface will be defined, as well as the necessary extensions to represent the required visualization on the VISO ontology.

4 SILKNOWViz Tool

4.1 Object Relationships

In the visualization of spatio-temporal data, the representation of different objects related to each other is one of the most complex activities. It is not possible to take advantage of the work carried out in the visualization of graphs, or trees, at least not in a general way, because in this model, location and time are fixed values, and therefore it is not possible to move them to improve their visualization. When there are many objects, the classic solution consists of developing clusters of objects, visualizing the relationship of an object with a cluster of objects, or relationships between different groups.

Flow maps [28] usually include trajectories. These visualizations systems have worked with this type of techniques in the relations between objects. This approach reduces the simultaneous number of connections and facilitates visualization and comprehension. However, in spatio-temporal data, groupings usually have a spatial base, and despite using these techniques, the number of connections to be visualized usually prevents a correct visualization. A typical solution is interaction: selecting an object, or group of objects, and displaying only the relationships of the selected object. An application that solves this problem by visualizing the objects relations interactively is shown in Fig. 5.

Fig. 5.
figure 5

Source: [28].

Visualizing all the relationships generates images that are very difficult to analyze. Visualizing the relationships of a single object is more readable.

Full size image

Using interaction has an important limitation: previous lack of knowledge whether an object is related or not, nor the number of its relationships. However, investigating those objects with an extreme number of relationships, different from the average, is generally more interesting. This problem becomes more difficult to handle when we want to study several elements simultaneously, as will be detailed in Sect. 2.

Is proposed the use of marks around the object, or clusters of objects, that identify the level of relations with the average. The mark would consist of a ring that would enclose the object, divided into as many sectors as objects it is related to. Longer or shorter lines will be drawn on this sector, depending on the number of relations. On the other hand, the length of the sector is associated with the time interval. Thus, the beginning of the sector will be the beginning of the period, and the end of the sector, the end of the interval. Each sector is a histogram that is enconded into a circle. A diagram showing the ring marker and the final appearance on a map can be seen in Fig. 6. This marker will be implemented in the tool and mapped using VISO.

Fig. 6.
figure 6

Ring mark for showing information about the number of relations that each object/cluster has with other objects/clusters during the selected time. To see the relationships you would have to select any of the objects. (Own image)

Full size image

The tool must present two implementations, because it is designed to be integrated in web pages, like a plugin, but also to run like a standalone application. One of the versions will be limited, displaying only the visualization zone. Another one will include all the features of the tool. The input of the application consists of access to the ontology, by means of a RESTFul API SPARQL, and a query to show the results. The query is optional because the queries may be constructed or extended from the tool itself, if the user requires so. The ontology will need to have the extensions VISO + RVL.

As far as the user interface is concerned, the tool has a display section and a query definition section. The display section displays the query results, allowing the user to change the point of view and select items.

The query definition section has the following elements:

  • A three-dimensional graph with the ontology classes that can be displayed, and their relationships.

  • A dynamic section with a form, whose data depends on the class selected in the graph, to specify its properties and filter the query.

  • A timeline control to set the time interval.

  • A window to visualize the selected elements properties.

There are two main scenes:

  • 2D Scene: a map of the region, Europe being the largest area, allowing you to move interactively and zoom in to city level. All the elements that appear in the map are two-dimensional, icons, clusters, etc. Levels of detail are available in this visualization. From a certain level, clusters are disaggregated and objects are represented with icons.

  • 3D Scene, within which there are three classes:

    • Map of the region, with the same features as the two-dimensional, but all elements are three-dimensional.

    • Hypercube. In this mode it is necessary to divide the time interval into parts (a maximum of four). A map of the region and the data, with a certain level of transparency, is displayed in each section of the hypercube. It is possible to select the different elements of each map of the hypercube. This technique requires interactive tools to examine the data.

    • Gallery. Is the top level of detail, only available for an entity with a collection of objects (for instance, museums). A three-dimensional virtual gallery with the objects is displayed.

A diagram of the different sections and display modes of the tool interface can be found in Fig. 7.

Fig. 7.
figure 7

At the top left appears an outline of the query definition area, the rest are representations of the different display modes. (Own Image)

Full size image

4.2 VISO Extensions

The VISO ontology and the RVL language form a very powerful set to define how the data of an ontology should be visualized, and to specify the interaction between them. It is a useful tool to define how to represent and interact with the classes and relationships of an ontology. However, it has two limitations for the purposes of our tool. First, since it is very generic, it lacks tools to define complex scenes, shaped by other graphical elements, in a specific order and position. Secondly, it is very oriented to representation in two dimensions, so it is necessary to create extensions for three-dimensional visualizations.

In order to solve these problems, the next extensions are created:

  • Creation of the 3D_Object class, which represents a generic three-dimensional object. Creation of the Scene class, which represents a scene, as well as relations with 3D_Object, to define the structure of the scene, following a philosophy similar to the one used by Swing. Is also defined the LOD (Level Of Detail) class, in order to manage the changes in function of the proximity of the viewer.

  • Extensions of VISO objects have been created, with 3D_Object as superclass. In this way, existing objects can be represented in three dimensions in a simple way. For example, the 3D_Map class will extend to 3D_Object and the Map class, which indicates that a Map has to be displayed. In this way, the 3D_Map class will indicate that this class has to be represented with a three-dimensional map.

  • Classes have been created for the special controls: Ring_Mark, Hypercube and Timeline.

5 Conclusions

In this work, on the one hand, we have reviewed the state of the art in tools for displaying spatio-temporal data. On the other hand, the use of ontologies in the visualization of this type of data and its relationship with Big Data has been analyzed. In addition, ontologies that depict how to visualize and interact with their own data have been reviewed.

The development of an open source tool has also been described, as a way to solve part of the existing problems in these lines of research and their practical application. This tool is designed for the visualization of spatio-temporal data contained in an ontology.

The most important details of this tool, as well as the user interface, have been explained. This open source tool is based on the use of the VISO ontology and the RVL language. This ontology has also been extended, to satisfy the tool requirements.

This tool will be fully developed during 2019 and it will be evaluated by a set of experts in cultural heritage from France, Italy and Spain. It will be available at https://github.com/silknow/silknowviz.