Keywords

1 Introduction

1.1 Visual Working Memory and Cognitive Load

Visual working memory (i.e. VWM) supports high cognitive functions providing temporary storage for retained information from one fixation to the next. The visual features primarily maintained in mind are position, shape, color and texture of the objects in space commonly referred an object’s attributes above the perceptive threshold  [1]. The number of items stored in memory is rather related to the concept of capacity  [2] and limited to 3–4 multi-attribute objects  [3] depending on subjective performance and task characteristics (for example, encoding time  [4]). Multitasking impacts the number of memory items that can be maintained and the amount of cognitive resources expended  [5]. Indeed, working memory not only includes maintenance of information, but also information processing during encoding time of filtering irrelevant stimuli (distractor avoidance)  [6]. Such multiprocessing is commonly termed the cognitive load. The time-based resource-sharing model  [7] aims to theorize the relationship between cognitive load and memory performance by identifying four major mental stages: encoding, filtering distractors, recall, and refreshing-the last of which redirects attention to restore dwindled items into memory. While learning during a classroom lesson, students need to manage mental resources in order to continuously acquire incoming information and effectively manipulate it. When information to be held oversteps working memory capacity, cognitive overload is experienced  [8]. Naturally, what happens during the classroom experience could be extended to other real-world scenarios. In multimedia, visual and audio streams are processed in humans by two separate brain circuitry both with limited working memory capacity, thus active processing of both streams could exceed the audience’s available cognitive capacity. This is not only due to the essential processing of audio or video materials, but also caused by environmental disturbances or confusion in the presentation of multimedial contents  [9].

1.2 Neurophysiologic Correlates of VWM

In literature three components are commonly attributed to VWM. Two of them are event-related potentials (i.e. ERP), one called controlateral delayed activity (i.e. CDA) and the other N2pc. The third correlate of VWM is an induced modification over occipito-parietal electrodes in alpha band, where one observes a decreased alpha power when an individual retains information in memory.

Lateralization of Stimuli in the Visual Hemifield. During VWM experimental evaluation, an event related potential called controlateral delayed activity (i.e. CDA)  [10] was identified on posterior-occipital areas usually interpreted as a neural marker of the number of items stored in memory. Moreover, CDA polarity changes depending on the position of the elements memorized in the visual space. Despite an alternative hypothesis claiming a relationship between the CDA wave and spatial attention  [11], current findings confirm the relation between CDA and memory capacity  [12, 13]. In subjects with lower memory capacity, distractors presented alongside targets in the visual field increased CDA amplitudes compared to individuals with better capacity  [14].

Deployment of Visual Attention. ERPs can track shifts of attention to the targets: the N2pc wave specifically reflects the attention towards an object  [15, 16]. This wave appears between 180 and 300 ms (usually peaking at a latency of 250 ms) with enhanced amplitude over the posterior-occipital electrodes contro-lateral to the visual targets present in the visual field. In  [17], the authors describe a relationship between N2pc and the processing load: visuo-spatial configurations that require more time to be evaluated demand a sustained involvement of attention reflected by higher N2pc amplitudes. N2pc appears not only as a metric of attention for targets in the extra-personal space, but moreover N2pc amplitudes are modulated during memory retention to weigh information according to its relevance  [18].

Relation Between Alpha Oscillations and Memory Functions. Alpha oscillatory activity is suppressed during memory encoding likely due to visual processing, whilst alpha is enhanced in the course of memory maintenance to prevent competition for resource allocation with incoming visual stimuli  [19]. Throughout memory retention alpha waves could phase-couple with other frequencies like beta or gamma in organized neural networks  [20]. Patterns of alpha desynchronization during encoding and synchronization for memory maintenance appear in form of induced power changes that are not time-locked to stimuli  [21].

2 Methods

Experimental Paradigm. Data was released by authors of the paper  [22] in pre-processed format. The present study solely retained the healthy subjects, originally denoted as the comparison group. Participants were 27 college students (11 females) with an average age of 22 years. The experimental paradigm was adapted from  [14] with an initial presentation of an array of geometrical shapes succeeded by a memory maintenance period of one second. Prior to each trial, an arrow appearing at the center of the screen informed the subjects which visual hemi-field they should memorize (left “L” or right “R”), while during each trial an array of colored shapes was displayed for 200 ms. Subsequently, the object array disappeared, requiring the subjects to store the visual information into their working memory. Following a memory retention period of one second, participants would then be tasked with deciding whether the new object array matched the one presented prior. Half of the trials had one color of a shape changed in the attended side. Three types of object arrays were tested: “low memory load” (i.e. “L”) with only 2 squares to be memorised, “high memory load” (i.e. “H”) when 4 squares were to be memorised or “distractors” (i.e. “D”) with two circles that should be ignored (Fig. 1).

Fig. 1.
figure 1

Experimental stimuli were 2 types of objects (squares and circles) in 10 randomly selected colors (supra-threshold perceptive features)  [22] (Color figure online)

Full size image

Pre-processing of Neural Signals. EEG was recorded with a high-density cap of 128 electrodes (Hydrocel Geodesic sensor net) and sampled at 500 Hz. Epochs were 104 for each experimental condition, and those with artifacts were rejected. Further pre-processing steps included filtering (0.05–30 Hz) and re-referencing in common average mode. Epoch length of the memory retention period was 1 s preceded by 0.4 s of baseline. For each subject, extra-cephalic electrodes (lower line of the Geodesic cap) were discarded (109 channels kept), and single trial EEG signals had their voltages normalized with z-score procedure (using baseline mean and standard deviation in range −200 to 0 ms): z-score standardization on subjective baseline allows one to model a user-independent approach. Procedures involved employed the MNE-python library  [24].

Outliers Detection. Outlier detection is an important preliminary step to identify subjects that the model is unable to generalise, deviating significantly from the rest of the data. Multidimensional scaling reduces dimension of the data by representing the proximity (or similarity) matrix between individuals as a lower dimensional space  [23]. The proximity matrix is a configuration of points in Euclidean space such that the inter-point distances approximate the subjective data. In (Fig. 2) dashed lines represent the squared Mahalanobis distances of the empirical covariance matrix shown as a visual indicator of adjacency between points. The inclusion criteria implemented a mathematical approach to cut-off outliers by identifying a critical threshold based on the approximated F distribution  [25] (Fig. 3). Based on the critical threshold, subjects 4 and 16 were excluded from further analysis.

Fig. 2.
figure 2

All subjects in all conditions

Full size image
Fig. 3.
figure 3

Outliers critical threshold

Full size image

3 Results

Initially, a total of 5956 signals were further normalized with L2-norm and pair-wise comparisons between conditions obtained with t-test analysis for each channel with significance threshold adjusted by Bonferroni correction. This statistical test was applied to identify a group of electrodes with significant changes between conditions. Indeed in Fig. 4, one observes a high number of differing pair-wise comparisons between 192 and 268 ms. Additionally, an analysis using cluster permutation F-test was conducted in the same data to highlight time frames of relevant activity differing between stimuli, as in  [26]. For example, electrode E90 in Fig. 5 illustrates the cluster of significant activity corresponding to N2pc time range (highlighted in orange, p < 0.05). Crossing the results of both statistical methodologies we obtained a group of electrodes and a time-frame with brain activity relevant to distinguish between conditions: mainly posterior-occipital channels during N2pc time course (Fig. 6).

Fig. 4.
figure 4

T-test on all sensor data

Full size image
Fig. 5.
figure 5

Permutation test on each sensor (Color figure online)

Full size image
Fig. 6.
figure 6

Topographic plots at 240 ms (color scale blue to red from −2.65 to 3.56 \({\upmu }\)V) (Color figure online)

Full size image

3.1 Cognitive Load Prediction

Analysis focused on prediction of three cognitive states using the time series extracted from the N2pc wave on parieto-occipital electrodes. Number of trials were equalized randomly under-sampling the most represented classes. From the time course of the N2pc, 794 features were extracted using 63 characterization methods  [27]. Redundant features were eliminated by the statistical approach presented in  [28], and divided in train and test sets with a 80% and 20% split. Test set accuracy is reported in Table 1 comparing two classification methods each with hyperparameters tuned by a combination of cross-validated (5 folds with stratification) randomized and grid search on train set. The chance level was calculated as 51.96% according to the binomial cumulative distribution as in  [29] with a significance threshold set at p = 0.05.

Table 1. Cognitive load prediction one-vs-one
Full size table

3.2 Prediction of Cognitive Load and Stimuli Location

In this investigation six classes were inspected: three cognitive load levels (“L”, “H”, “D”) by two different locations of the stimuli in the visual hemi-field (“L” or “R”). In Fig. 7, the global field power (i.e. GFP) from all electrodes is used to identify a time window enclosing the N2pc wave. Indeed, permutation t-test with pair-wise comparisons (significance adjusted by the “false discovery rate”) between conditions identified two clusters of significant electrodes, one over the parieto-occipital, and one over the fronto-central areas (Fig. 8). Area of the N2pc from each single trial over the identified electrodes was used as feature for an SVM classifier. Classes were balanced by randomly under-sampling trials of the most represented labels (5346 trials used as observations). A support vector machines (i.e. svm) model with radial basis function (i.e. rbf) as kernel was selected as classifier and data was divided in the aforementioned 80–20 split. Kernel and regularization parameters were optimized by grid search with cross-validation (5 stratified folds and validation size 20%) and classification methodology one-versus-rest. Outcomes against a dummy svm-rbf classifier are shown in Table 2 using F1 score as metric (last two columns are weighted-average F1 and micro-F1 scores). Statistical chance level calculated as in  [29] was 17.5% (at p \(=\) 0.05).

Fig. 7.
figure 7

GFP of the experimental conditions

Full size image
Fig. 8.
figure 8

38 identified electrodes

Full size image
Table 2. Prediction of cognitive load and stimulus location
Full size table

4 Conclusions

Two procedures are presented to distinguish between the levels of cognitive resources deployed during retention period of the visual working memory: the first based on features extracted from the N2pc time series whereas the second involves using the area of N2pc component. The latter offers a promising technique for categorizing not only the memory load, but also the location of the stimuli in the visual hemi-field (overall accuracy +38.5% above chance level).