Understanding multi-modal brain network data: An immersive 3D visualization approach

Highlights

• We developed a novel concept for 3D VR visualization of multi-mode EEG brain connectivity networks comprising multiple features within one view.

• The anatomical arrangement of network nodes, together with color-coded temporal evolution along directed links between the nodes offers an intuitive understanding of network patterns.

• Users can interactively explore brain networks in VR, leading to a higher motivation and offering the possibility of a non-hypothesis-driven research.

• According to a standard system usability scale questionnaire, experts from neuroscience, as well as from the field of computer science are convinced by the system usability.

Abstract

Understanding the human brain requires the incorporation of functional interaction patterns that depend on a variety of features like experimental setup, strength of directed connectedness or variability between several individuals or groups. In addition to these external factors, there are internal properties of the brain network as for example temporal propagation of connections, or connectivity patterns that only occur in a distinct frequency range of the signal. The visualization of detected networks covering all necessary information poses a substantial problem which is mainly due to the high number of features that have to be integrated within the same view in a natural spatial context.

To address this problem, we propose a new tool that transfers the network into an anatomically arranged origin–destination view in a virtual visual analysis lab. This offers the user an opportunity to assess the temporal evolution of connectivity patterns and provides an intuitive and motivating way of exploring the corresponding features via navigation and interaction in virtual reality (VR). The approach was evaluated in a user study including participants with neuroscientific background as well as people working in the field of computer science. As a first proof of concept trial we used functional brain networks derived from time series of electroencephalography recordings evoked by visual stimuli. All participants gave a positive general feedback, notably they saw a benefit in using the VR view instead of the compared 2D desktop variant. This suggests that our application successfully fills a gap in the visualization of high-dimensional brain networks and that it is worthwhile to further follow and enhance the proposed representation method.

Introduction

The field of experimental and clinical neuroscience is constantly gaining importance [1]. While methods for the analysis of brain anatomy or brain-behavior relationships improve, large parts of the functionalities of the human brain are yet to be discovered. In particular, the investigation of directed information transfer within the human brain is a growing field of research. Understanding how the components of these complex neural networks within the human brain interact and affect each other is essential to understand and treat various different neurological, mental or developmental disorders [2], [3], [4].

Interaction and inter-connectivity of neurons or brain regions is called brain connectivity and can be separated into three categories [5]: structural connectivity refers to concrete neuroanatomical connections, functional connectivity describes the temporal correlations between neurophysiological events of spatially neighboring or remote neuronal structures and finally, effective connectivity is defined as the directed influence of one neuronal structure on another, mediated directly or indirectly. Amongst others, the networks can be derived from various neurophysiological recording techniques such as electroencephalography (EEG), magnetoencephalography, positron emission tomography and functional magnetic resonance imaging [6], [7]. Within the last few years, the combination of simultaneously recorded EEG and fMRI data has received growing interest in studying neural activity and, still more recently, also for the investigation of neural connectivity [8], [9].

In our work, we focus on functional connectivity derived from the analysis of EEG time series data. Here, it is assumed that the underlying functional brain connectivity patterns possess a certain strength and direction of neural information transfer. They are thus considered as so-called weighted, directed networks. Additionally, these networks contain information with respect to several modes. . The mode space indicates the location of possibly occurring directed connections within the brain (i.e. position of electrodes on the scalp). Consequently, this information includes the anatomical context of information transfer within the brain network. Temporal evolution of the functional connections is integrated via the mode time, which is of particular interest when it can be expected that the network is undergoing change in the course of a cognitive task. Finally, the mode frequency must be considered, too, as electrical brain activity is characterized by typical frequency ranges (also known as frequency bands) of brain activation that show different properties depending on a mental state or cognitive task. These frequency-dependent variations do not only occur in brain activation but also in functional brain connectivity [10]. Without any further processing, however, the understanding of resulting brain networks comprising all three modes at once is impossible. A graphical representation can offer a useful tool in this situation, yet the effective and intuitive visualization of such multi-dimensional data remains a major challenge. Current solutions mainly rely on (1) spatial context by showing aggregated or reduced data, or (2) crowded abstract visualizations [11], [12]. Regarding (1), aggregation leads to a loss of detail and the outcome depends on the parameters the data are aggregated upon (e.g. mean across time or mean within a distinct frequency band). Data reduction achieved by a preselection of displayed electrodes has the disadvantage that the overall view on the brain network as a whole gets lost. Regarding (2), abstract matrix-like visualizations can integrate time and frequency dimensions of the complex data, but they lose the spatial context making it less intuitive and more difficult to understand.

In order to counteract these limitations, we propose a visualization tool with the goal to fulfill the following five requirements R1, $\dots$, R5. The approach should

• allow a view on the whole weighted, directed network comprising all EEG electrodes,

• keep the anatomical context to offer an intuitive understanding of brain networks,

• visualize the propagation of connectedness over time,

• integrate various frequency ranges and

• additionally offer the possibility for the application of individual, case-specific restrictions (e.g. choice of certain brain areas or thresholds for minimum connectivity strength of displayed connections).

Finally, the general demand beyond all these requests must be to provide an efficient inclusion of all aspects listed above in order to make the overall perception of the complex visualization as intuitive and understandable as possible.

To achieve this goal we propose a novel visual analysis tool in a 3D immersive virtual reality (VR) environment. Electrodes are displayed in an anatomical arrangement across the human scalp and the temporal evolution of directed interaction is color-coded by the edge weights within the brain network. That means, in contrast to conventional visualization approaches, the color of the 3D links between brain areas changes along the length of the edges. Interaction opportunities like selection of distinct EEG electrodes or filtering via switching between different edge weight thresholds offer the possibility to adaptively visualize the data and enhance spatial perception of appearing patterns within the brain network. Notably the application offers an effective way of a non-hypothesis-driven exploration of brain connectivity: Regions of interest regarding any of the dimensions space, time and frequency do not have to be known a priori and can be explored via the immersive 3D view, helping to generate new hypotheses which had not been in the focus of neuroscientific research yet.

This work is an extended version of the conference paper “Immersive 3D Visualization of Multi-Modal Brain Connectivity” [13].