Keywords

1 Introduction

The notion that working memory is the basis for temporary storage and processing the complex information and it does effect the understanding ability of human was generally accepted ever since the seminal studies proposed by Baddeley [1, 2]. Studies have been completed to establish the high correlation between the working memory and the study ability [3, 4]. Other human’s cognitive abilities are much higher than the short-term memory storage, thus the working memory should be seen as a short board and the key factor of human’s cognitive ability. Due to this, numerous researchers were interested in the working memory.

The big data visualization nowadays with the characteristics of high dimensions and magnanimity is usually displayed in a dynamic form. Along with the development of computer hardware and software technology, the dynamic visualization could reach the close to real-time and presented smoothly. The smoother of the visualization, the better of it, is that reasonable? Since human is the cognition subject, the unique index for evaluating the visualization should be the human’s cognitive performance. Due to the complexity of big data, visual cognitionis a typical complex task usually accompanied with high cognitive load. Therefore, the visualization mechanism of large database should follow the characteristics of working memory to acquire better cognitive effects. In the experiments of our previous study [5], we noticed that the recall accuracies of the subjects were significant higher in the condition of temporally shield than the condition of skip gradually and the condition of skip directly without halt. This finding made us to pay attention to the relationship between human’ working memory features and the information’s visualization presentation.

According to the prior studies, working memory includes both the functions of processing and the storage [6, 3]. As for the process of working memory, one description was task-switching resource-sharing model [7, 8], while another was time-based resource-sharing model [9]. There was no essential difference between these two resource-sharing models. According to these resource-sharing models, processing and storage were competition for the limited working memory resources when we receive the novel information. As time progresses, attention switched from processing to the retrieval, refreshing of decaying memory traces was needed to recall the past items. However, previous studies were focused on the phonological loop subsystem (i.e., reading task) of working memory [10, 11]. And since the on-screen visualization is based on the graphical understanding, it involves more of the visuospatial sketchpad subsystem of working memory, our research was more focused on the visuospatial sketchpad subsystem. And the major goal of this study is to establish whether we need to empty our working memory storage in appropriate time during the displaying of dynamic visualization to let the novel information in.

Eye tracking data have already been used to measure human’s mental workload in real-time and accurately [12, 13]. The eye tracking measure is better than the traditional performance measure on reaction time or accuracy for its precise temporal reflection of mental process. It can indicate the time, space and duration data about the participants’ fixating on certain stimuli. Studies revealed that total duration time index performed a better predictor than other index like blink duration [14]. So, we choose the concurrent eye movement data total duration time and fixation number and the performance data reaction time to estimate the real-time cognitive load in this study.

2 Experiment

2.1 Subjects and Materials

The theoretical background and the results of the previous studies led to the following experiments to verify the validity of the halt in the dynamic visualization. The experiment was 2 × 2 between-subjects design with one experimental group and one control group. We designed two kinds of experimental material, as shown in Fig. 1. One was arrayed in dot matrix, while another was in annular, and both of them were common to see in data visualization. In the matrix figure, there was 1 target dot and 329 distraction dots, and 2 target dots with line connected and 90 distraction dots in the annular one. The matrix (labeled Vis. 1) was defined as hard task for more interferences in the figure and it’s hard to distinguish the similar target dot in it. And the annular (labeled Vis. 2) was defined as easy task since the distraction dots were much less than the matrix figure and the line connected the two target dots made the subjects identify and judge easier. The target of the searching task was a solid polygon with approximate circular shape while the interfering objects were circular solid points, seen in Table 1. The meaningless of both the target graphic and the distractors could prevent the subjects to use the image thinking during the searching tasks. The subjects were asked to reaction by press the keyboard as soon as they found the target node.

Fig. 1.
figure 1

Two kinds of experimental material (a) was arrayed in dot matrix, the corresponding searching task was hard; (b) was arrayed in annular, the corresponding searching task was easy.

Full size image
Table 1. Interference objects and target object of the material in both the experimental and control group
Full size table

And 21 graduate students participated in each group. The Tobii X2-300 compact contactless eye tracker was used to collect the eye movements’ and performance data. The instrument sampling frequency is 30 Hz, the staring accuracy is 0.4°–0.5° and the head movement range is about 50 × 36 cm. The resolution ratio of experimental animation material is 1280 × 960 px. The experimental materials were presented by a HP 21 inch screen with the brightness of 92 cd/m2. The laboratory was in the normal lighting conditions (40 W fluorescent). The distance between the subjects and the screen was about 550–600 mm.

Since there was no exact value about the visual short-term memory retention [15, 16], a 700 ms interval of blank screen was setup before each visual graphic in the experimental group based on experience. Both the experimental group and the control group tested two kinds of figure three times (including the trials with homogeneous figure). The only difference between the experimental group and the control group was the 700 ms interval before the figure presented in each trial.

2.2 Procedure

The materials presented in experimental group were as same as in the control group. The procedure of the test trial in each experimental group was showed in Fig. 2. The top row was a test trial in the control group, and the below row was a test trial in the experiment group with a 700 ms blank interval before each test picture. The subjects were tested repeatedly in six homogeneous trials, and in each trail the location of the target point appeared randomly in each test trial while the form of the materials keeping consistent. In order to avoid the effect of the attention blink on the experimental results, a blank picture with “+” was displayed for 500 ms to eliminate the visual residue [17]. The reaction time was recorded from the moment when the picture displayed in the screen to the moment when subjects pressed the keyboard as soon as they found the target.

Fig. 2.
figure 2

The procedure of a test trial in each group. The top row is a test trial in the control group, and the below row is a test trial in the experiment group with a 700 ms blank interval before each test picture.

Full size image

3 Results

The two-factor ANOVA was used to analyze the results with the two factors of task difficulties and blank interval. The response time (RT), the number of fixation time (N) and the total fixation duration (FD) were taken as the dependent variables. Among them, searching time data RT represented the searching performance level and the concurrent eye movement data N and FD could reflect the subjects’ real-time workload situation. As to estimate the situation of performance and eye movement under a certain workload, the data of the first trial in each group were not counted in. The results showed that the average values of searching time, number and the total duration of the fixation were relatively higher in the condition of hard task (Vis. 1) and the condition in control group (without blank intervals). As seen in Fig. 3, the lines of reaction of three factors RT, N and FD had the intersecting tendencies with strong consistency. It implied that the influences of the blank interval were different in the conditions of different task difficulties and the lower performance and the higher real-time workload in the situation when the task got hard or the blank interval was omitted.

Fig. 3.
figure 3

The interaction appeared in the dependent variables of RT, N and FD.

Full size image

In particular, seen from the results of pairwise T-test analysis between the control group and the experimental group (Table 2), there were significant differences in RT (t = 2.163, df = 41, sig. = 0.036 < 0.05), N (t = 2.124, df = 41, sig. = 0.040 < 0.05), and FD (t = 2.174, df = 41, sig. = 0.036 < 0.05) in the condition of hard task. And these differences were unidirectional, that were, the performance level increased (RT decreased) when the blank intervals were added. And no significant differences existed in the condition of easy task, with the results of RT (t = −0.827, df = 41, sig. = 0.413 > 0.05), N (t = −1.025, df = 41, sig. = 0.312 > 0.05), and FD (t = −0.847, df = 41, sig. = 0.401 > 0.05).

Table 2. The pairwise sample T-test results between the control group and the experimental group with the dependent variables RT, N and FD
Full size table

Besides the value of concurrent eye movement data fixation number and total fixation duration, the difference of real-time workload in different tasks could be seen clearly in the heat map, seen in Fig. 4. The top row (a, b) was the heat maps of the saccadic data of a typical subject in experimental group (with halt); the bottom row (c, d) was the heat maps of the saccadic data of a typical subject in control group (without halt). Figure 5 showed the overlapped heat map of total 21 subjects in each group. As same as Fig. 4, the left two maps (a, c) showed the eye-movement heat maps generated by the same experimental figure. We can see the obvious differences in the saccadic number and path between the left two maps (a, c), and these differences hardly be found in the right two maps (b, d). It showed whether there was a halt to be significantly affected the eye movement in the condition of hard task, and no significant difference in eye movement in the condition of easy task, due to the low level of overall workload.

Fig. 4.
figure 4

The heat maps of two selected subjects of the experimental group and the control group. The top row (a,b) is the heat maps of the saccadic data of a subject in experimental group (with halt); the bottom row (c,d) is the heat maps of the saccadic data of a subject in control group (without halt).

Full size image
Fig. 5.
figure 5

The overlapped heat maps of 21 subjects of the experimental group and the control group. The top row (a,b) is the overlapped heat maps of the saccadic data of 21 subjects in experimental group (with halt); the bottom row (c,d) is the overlapped heat maps of the saccadic data of 21 subjects in control group (without halt).

Full size image

We can see the different searching preferences from the figure of eye movement track. In the trials of matrix figure, subjects always scanned the figure with the habit of reading words. They scanned from left to right and from top to bottom generally, than they wandered around the suspected target to make sure the answer. In the trials of annular figure, most of the subjects browsing along the circle (counterclockwise majority), than stayed around the suspected targets to determine. In Fig. 5(c),it was hard to see the underlying experimental figure since the gaze points almost distributed on the entire figure. The circular motion of the eyeball was much easier than the repeat horizontal movement. It was one of the reasons that mean reaction time in the annular figures was shorter than in the matrix figures.

In addition, all the Pearson correlation coefficients between any two factors of RT, N and FD in each trial were greater than 0.90 and reached the significance at the 0.01 level. This result strongly proved the high degree of correlation between the performance and eye movement. Since the concurrent eye movement index reflected the instantaneous cognitive load, the high correlation between operational performance and cognitive load in the interface task was confirmed again.

4 Discussion

The reason of the results should be attributed to the limitation of working memory span. Based on the previous studies, working memory fulfills the functions of processing and storage at the same time. When the novel information flows into working memory and the amount exceeds the span restriction of the working memory (e.g., in Vis. 1), the processing performance would be undermined. In this study, the processing task was to judge the point to be the target or interference one. And if the tolerance of the novel information were within the span limited of working memory (e.g., in Vis. 2), the processing performance would not be influenced. The experimental results showed that the storage and processing processes competed for limited attention resources when the memory span demands for the current task excessed what it can tolerate. Since the experimental materials in this study were abstract graphic, which were difficult to convert to verbal memory, the working memory involved in this study was Visuospatial Sketchpad subsystem. And it exhibits the same span limitation as same as the Phonological loop in the previous studies.

Therefore, we can explain the results as the following. When the visual complexity is low, the halt has no significant influence in the cognitive load and the searching performance. While the visual complexity getting higher, the appropriate halt has a positive meaning to reduce the overall workload and improve the search performance. Based on the characteristics of big data visualization, the visual materials which we see in our real-life situation tends to be much more complex than in the experimental conditions (since all the factors of texts, graphics, background, etc. that could increase the overall cognitive load have been removed) in this study. So we can reasonable assume that the dynamic data visualization in reality should be in a complex task condition.

The fact that the interval time before each searching task could reduce the cognitive load and improve the searching performance implied a positive meaning of the halt. So we need not pursue a completely smooth and interval-free dynamic visualization and the appropriate halt is beneficial to the human’s understanding. Maybe we need to set a reasonable pause or blank before the query information appears on the path of infinite pursuit of fluid visualization. We should halt for a while and consider people’s ability to accept. As we know, visualization is the bridge between the human and the data. The best visualization of data should be the most suitable one for human to read and understand.

In this paper, the definition of complex and simple task was too simple, we need to further define the task complexity to obtain more accurate conclusions. However,this research makes sense with the dynamic visualization design to some extent. Follow-up research of this study is how to set the halt in a visualization to make it not abrupt but friendly.

5 Conclusion

The spatial organization form of visualization objects could affect subjects’ visual search path. The nodes arranged in annular would result in higher search performance than in progressive sort. Due to the span limitation and the competition between storage and processing in human’s working memory, in the complex cognitive task such as reading big data visualization which inevitably with a high cognitive load, the appropriate pauses and blank would be beneficial to cognition in the course of displaying dynamic visualization. And in condition of simple cognitive task, the halt in the dynamic display is not necessary.