Keywords

1 Introduction

Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) are regarded as the new frontier in digital communication media and use of these new technology tools is rapidly increasing [1].

The development and increased availability of advanced brain measurement methods allows researchers to explore human behaviors and minds in different ways than were previously available. Researchers do not have to rely on entirely on the results from self-reported measurements, which have known limitations [2,3,4] or behavioral measures only. The emergence of neurophysiological measurements has opened up new fields and allowed others to grow because the tools allow us to explore previously unanswerable questions. For the reasons above, neurophysiological research has become increasingly popular.

This has been possible because the brain can be directly observed by non-invasive electrophysiological recording and the data shows accurate and high-resolution data.

2 A Review of Functional Neuroimaging and Electrophysiological Measurement

A cognitive neuroscience-based measurement is an advanced and improved technique to expand our knowledge of human behaviors and minds. These measurements are designed for non-invasively interpreting brain functioning at high temporal and/or spatial resolution. By capturing electric fields on the surface of the head induced by neuronal activity in the brain, we can specify human brain function.

This concept has been developed with the trust that the brain mainly controls physical and psychological activities [5]. For instance, this basic concept has widely contributed to develop brain-computer interfaces (BCI) using the brain as an addition input device for broader users including disable patients [6].

In addition, psychophysiological approaches to the brain have contributed to explore human minds deeper and wider without getting direct responses from participants. For example, neuroimaging studies have proven that increased amygdala activity when viewing fear [7], sad, angry [8] or happy facial expressions [7]. In addition, sensors are also able to measure mental workload [9], which is more effective and accurate than estimating workload from post-test questionnaires. The reason is first, it does not instantly reflect the result at the time of exposure (delay in response). In most cases, the questionnaire is completed after exposure, not during exposure of stimuli. Therefore, to complete a post-test questionnaire, a participant needs to recall the time of the experiment. Secondly, the result from the 5 or 7 scale questionnaires does not represent the details of participants’ minds and thoughts because human’s psychological process is not readily available for introspection. On the contrary, the brain imaging tool using a psychophysiological method continuously captures user’s cognitive process on a millisecond scale. For these reasons, psychophysiological measurements have been widely used even though it requires more time and cost to set the experimental environment up.

Then, what types of brain waves we can measure and what do they represent? There are technically different methods to measure brain waves, and each method has a different approach to explore human minds.

There are several physiological measurements (i.e. PET, fMRI or TMS), however we discuss only methods that measure brain waves with a head cap and sensors since a purpose of this paper is to suggest a new design of the cap for brain wave measurement system.

2.1 Electroencephalogram (EEG)

In contrast to other physiological measures such as heart rate, skin-conductance level, or facial electromyography (EMG), Electroencephalogram (EEG) is a direct measure of central nervous system activity from the active brain. In particular, the EEG can see the changes in electrical discharge of cortical neuronal populations; it measures the capacity or performance of cortical information [10]. In other words it can measure the degree of brain concentration [11]. It can measure the arousal dimension of human emotions from peripheral signals [12] which may include alpha, theta and/or frequencies greater than 16 Hz [13]. In addition, it is ease of use and low set-up cost. It generally uses from 2–256 channels to see each area of the brain by wearing a head cap and putting measuring sensors (electrodes) into grommets over the cap. Grimes et al. showed 99% accuracy in working memory states (WM) and four WM states with up to 88% accuracy with EEG [14]. Because of this accuracy, military also uses EEG to monitor pilots’ mental states while they are in the air [15] (Fig. 1).

Fig. 1.
figure 1

(Source: Biopac Systems)

EEG head cap

Full size image

2.2 Functional Near-Infrared Spectroscopy (fNIRS)

Functional near-infrared spectroscopy (fNIRS) is another advanced neuroimaging technology for mapping the functioning human cortex. It was originally designed for medical use and has been first clinically used in the mid-1980s. However, it is now playing an important role in both neuroscience and communication research [16]. The fNIRS uses a measurement of concentration changes in both oxygenated and deoxygenated hemoglobin (Hb). It is also a non-invasive imaging method to see the changes in blood oxygenation, that can represent levels of brain activation while EEG captures electrical waves associated with the activation potential of neurons. The fNIRS uses optical fibers placed on the scalp that send light in the wavelength range of 650–850 mm in to the targeted area on the head, where the infrared light is re-emitted based on the amount of oxygenated and deoxygenated hemoglobin from the issue during brain function from multiple channels (up to 256).

This method has been proven by many researchers as a valid method to measure hemodynamic levels originating from prefrontal cortex (PFC) activation [17] in emotion induction [18] and cognitive functions such as problem solving or memory related to mental workload [19] (Fig. 2).

Fig. 2.
figure 2

(Source: Hitachi Medical Systems)

fNIRS head cap

Full size image

3 Benefits of Psychophysiological Measures to Understand New Media

The brain imaging tools we discussed above have several common advantages for research. First, they accurately measure human mental states or workload with sub-second or even millisecond level precision. Secondly, those are popular, non-invasive and widely proven techniques in research. Third, EEG or fNIRS can measure brain waves in more ecologically valid conditions than other sensors. For example, using a PET or fMRI require participants to be stationary or lie in restricted positions.

For those reasons, EEG and fNIRS have been widely used in human computer interaction (HCI) studies, including usability testing and user-centered design for better human performance [9]. In addition, EEG and fNIRS are also suitable for new media research when scientists are interested in the cognitive mechanisms behind media effects [20, 21].

3.1 Virtual Reality

The concept of Virtual Reality (VR) is to experience a certain unreal environment as real. With the technological development in graphics, it is possible to depict a realistic environment in 3D. By wearing Head-mounted display (HMD), it is possible for a user to be in a virtual world without seeing other real objects near the user that may distract experiencing immersive VR.

In addition, other functionalities that increase users’ emersion, such as head-tracking or motion-tracking system, make a virtual environment more realistic so that users feel “they are actually there” [22,23,24]. Therefore, in recent years communication researchers and sociologists have studied to see their minds when they interact with other objects or people in VR [25, 26].

3.2 Challenges in Recording EEG and fNIRS in Virtual Reality

As we discussed above, it is essential to see users’ cognition when they interact in VR. It is essential to see how users accept and process information they get and interact with in virtual environments.

However, combining VR system and EEG or fNIRS is difficult because the HMD mounts interfere with the probes of the devices. There were some studies that see some social effects of virtual reality with electroencephalography (EEG) or fNIRS previously [27, 28], however they just see the effect of virtual object shown in a 2D display, not an HMD.

As shown in the Fig. 3, both HTC VIVE and Oculus Rift use a 3-axis headband for a perfect fit. In addition, the headband should be tightened enough to cut off the light from outside of the HMD, so the user can fully focus in virtual environment. However, once the user puts the brain measurement cap on, they are unable to wear the HMD over the cap since the cables and sensors attached to the cap block placement of the HMD. Since the HMD head bands are made with a flexible rubber, it puts pressure on the sensors. This pressure displaces the probes, which results in poor sensor contact with the head and consequently, poor data quality. Also, to measure prefrontal cortex activity in VR, the HMD cannot be worn correctly because the HMD obstructs the area over the prefrontal cortex. If the HMD is worn below the cap, then the HMD lens and eye are not aligned and centered, that finally results in a failure of setting up the virtual environment. Figure 4 shows the experimental setup of fNIRS. As shown in the picture, it is impossible to wear the HMD over the cap because of cables.

Fig. 3.
figure 3

(Source: Oculus and HTC)

Virtual reality head-mounted display: Oculus Rift and HTC Vive

Full size image
Fig. 4.
figure 4

Example of brain measurement setup

Full size image

For those restrictions, it has always been a difficulty for scholars to research an immersive virtual environment using sensors like the fNIRS.

3.3 Previous Solutions Suggested by Other Researchers

Figure 5 shows a method to use fNIRS in VR setup suggested by Seraglia et al. [29] in 2011. They developed a VR cap that does not interfere fNIRS cables. However, the configuration limits the regions of the brain that can be measured, which functionally prohibits research on topics like cognitive load. For instance, prefrontal cortex activity seeing many higher cognitive functions [30] cannot be measured with this setup. Many researchers investigated prefrontal cortex activity in the simulated environment created with computer graphics [31, 32]. Therefore, it was highly needed to develop a new brain measurement cap to solve this limitation.

Fig. 5.
figure 5

Suggested solution to use fNIRS in VR by Seragila et al. [29]

Full size image

4 M.I.N.D. Brain Cap, Measurement for VR Research: Design and Iterations

4.1 M.I.N.D. Brain Cap, Initial Design

To solve issues and restrictions of brain measurements in VR discussed above, we designed an innovative head cap that makes it possible to wear the head-mounted display over the brain measurement cap (Figs. 6 and 7).

Fig. 6.
figure 6

New design of the brain measurement cap for immersive virtual reality system (patent pending)

Full size image
Fig. 7.
figure 7

Side view of the new brain measurement cap

Full size image

There are several advantages to this design. First, researchers do not need to limit the number of channels to use because the cap is designed to make most channels available in the VR experimental setup.

Secondly, this cap saves time in setting up an experimental environment because subjects need to wear only one cap that incorporate (1) the HMD, and (2) the brain sensor array. In a previous setup, a subject wears the brain measurement cap first and then wear the HMD. Third, the cap design works with both EEG and fNIRS. Senor sockets can be easily replaced for each experimental measurement (Fig. 8).

Fig. 8.
figure 8

Back side view of the M.I.N.D. brain measurement cap

Full size image

However, there are also limitations of the cap. First, even though the main material of the cap is made of rubber, expansion of the cap could not cover all head sizes. For instance, optical signals showed poor in the front forehead area when the cap was worn on a smaller head. Secondly, connectors to attach HMDs are not universal, therefore we need to design a cap for each HMD (e.g. Oculus rift version, or HTC VIVE version). Thirdly, modifying the channel configuration and probe placement is difficult.

4.2 M.I.N.D. Brain Cap: Second Version

To solve the issues of the first version of the cap, we made several design modifications. First, we choose a modular design to address the size issue of the cap.

As shown in Fig. 9, the cap consists of three different parts and each part can be easily connected to other modules (probe holders) with several plastic clips. By doing so, a researcher can easily find a perfect fit (size) for participant’s head by configuring different size of cap modules. Therefore, we have a higher probability of accurate probe placement and better-quality data. Secondly, the researcher can easily rearrange the measurement area by choosing various size of modules, which is a standard option for non-VR research. For instance, standard configuration of the modular consist of 4 × 4 and 3 × 3 channels. If a researcher wants to concentrate on a certain area, then he or she can use 4 × 4, and choose 3 × 3 for the rest area in fNIRS (See Fig. 10 for details).

Fig. 9.
figure 9

Second version of the M.I.N.D. brain measurement cap (modular design)

Full size image
Fig. 10.
figure 10

Various sizes of probe holders (source: Hitachi Medical Systems)

Full size image

5 Conclusion

The emergence of VR and AR as a readily available communication tool and media interface has pushed researchers to begin exploring the cognitive neuroscience associated with VR/AR experiences. However, conducting studies is difficult because of technical challenges associated with integrated HMDs and sensors. The M.I.N.D. brain measurement cap is the first attempt at addressing these problems; we hope our design makes it easier for researchers to combine VR/AR and EEGs and fNIRS.