Keywords

1 Introduction

To determine the neuronal behaviour elicited on the spectator by such an event as a shot change in any feature film presents more challenges than if one were to use audiovisual stimuli created ad hoc for isolated laboratory studies. However, being able to analyse real cases instead of cases designed for a particular study, allows us to get closer to the viewer’s neuronal response to the reality of a film. This kind of research could open the doors to the introduction of new kinds of biofeedback in the emerging field of Adaptive Storytelling.

The cognitive study of shot changes in films has been approached from fMRI [1], ERD/ERS [2] and blinks [3] among others. Usually studies based on ERD/ERS use material specifically made for laboratory studies. The aim of this study is to analyse the ERD/ERS responses to cuts in real film fragments. To achieve this an adaptive procedure is developed to effectively locate areas of the ERD/ERS that coexist across cuts to avoid locating ERD/ERS areas produced by a stochastic kind of cut. To obtain this analysis, we needed to identify reactions to different types of cuts within different aesthetic contexts.

The development of brain-computer interfaces allows a new form of evolution in the cinematographic discipline, for example in the emerging field of adaptive storytelling and the interactive cinema [4,5,6], but in order to have a deep control of the interaction it is necessary to break down how the cinematographic techniques affect the cognitive system of the viewer. The first human-film iterations have already appeared where the viewer intervenes in the edition [7], but there is still a great need for evolution to achieve a full brain-computer iteration that determines the interaction with the film.

To accomplish the objective of locating patterns of neuronal reactions to the shot change event through the ERD/ERS, we developed a methodology that combines the permutation test and Spearman’s correlation test on a six-sample size sliding time windows of the power change for every pair of model signals. When a specific time window for a specific frequency band and electrode appears correlated in all possible combinations of pairs, we can consider that we have located a neuronal reaction in ERD/ERS that is responding to the cuts.

2 Previous Researches

Among the previous studies on cinematographic technique through the ERD/ERS, it is worth mentioning those conducted by Heimann [2, 8] and Martín-Pascual [9], due to the similarities with the present investigation. Different from the present studies, they all created their own filmed material to develop their experiments. They try to replicate in the audiovisual fragments the same conditions in each clip introducing the minimum number of possible changes. So, in the work «Moving mirrors» [8] they explain how they filmed the same situation with different camera approach techniques, trying to keep the rest of the inputs constant in the different shots. To be able to analyse the viewer’s neuronal response to a specific cinematographic technique in the most concrete possible way, Heimann introduced differences only in one variable in the filmed audiovisual material that was shown to the viewer.

However, in the present study, we set out to work with existing feature films, so we face the challenge to isolate the input of the cut of all the non- related features that appear in the film after the shot change by cut. In this way, we consider that we can perform an analysis of the film edition closer to the real cinematographic experience than working from material filmed ad hoc for laboratory purposes.

3 EEG Registration

The experimental designed consisted on the selection of four film fragments belonging to commercial films together with the design of the electroencephalographic record. The film fragments contain the shot changes and represent different cinematographic styles in order to have the widest possible representation of reactions due to different technical, aesthetic and stylistic characteristics of the films. Subjects who participated in the experiment visualized the film fragments while we recorded their electroencephalogram. The selected film fragments belong to Bonnie & Clyde, The Searchers, Whiplash and On the Waterfront.

Twenty-one subjects participated as spectators. The participants did not report any neurological disorder, psychological problems or being under medication. The subjects were kindly asked not to consume exciting substances (such as coffee) or depressants (such as alcoholic drinks) that could modify cognition in the hours before the experiment.

The EEG signal from the 21 volunteers was recorded using a 31 electrodes array distributed over the scalp following positions defined by the international convention of the American Electroencephalographic Society in its 10–20 system for EEG-MNC [10] (Fig. 1).

Fig. 1.
figure 1

Distribution of the electrodes following the international system 10–20 for EEG-MNC.

Full size image

Electroencephalographic registers were made between 15 and 30 of May 2017. The four film fragments were played to each subject through the Unity game engine [11], while the electroencephalogram was registered in MatLab though two different computers synchronized by means of a User Datagram Protocol Network (UDP).

4 Signal Analysis Design

In order to identify neuronal patterns triggered by the event of the cut in the EEG signal we divide the analysis in two phases. The first phase consists in preparing the signal registered by the EEG for the study in the frequency domain. The second phase consists in the statistical analysis in the power change of the signals obtained to locate reaction patterns produced as a result of the shot change by cut.

4.1 Signal Preparation

After recording the electroencephalogram an ICA (independent component analysis) was performed in order to isolate for each electrode the signal from its registration area, discarding the interferences from other electrodes recording areas. Then, we proceeded to perform a manual cleaning of artefacts from the recorded signals. From there, we generated one model of neuronal reaction signal for each of the audiovisual fragments yielding four models. To do this, we divided the continuous signal recorded for each user into the four sections of interest corresponding to the recording of each audiovisual clip shown, one for each film, eliminating the rest of the signal. Using the MatLab G.bsanalyze toolbox, we averaged the signals for each electrode in each audiovisual fragment for all 21 subjects [12]. Therefore, we ended up with a set of four model signals, one for each of the audiovisual clips and for each of the 31 electrodes averaged across the 21 spectators analysed. The scheme of the process of creating the model signals is shown in the Fig. 2.

Fig. 2.
figure 2

Scheme of signal registration.

Full size image

Given that the regions of interest for our study are the instants after the cut, from each model we selected only the 2000 ms time intervals centred in each cut (one second after and one before) and dismiss the rest (left part of the Fig. 2). In this way, we ended up with 84 two-second epochs for Bonnie and Clyde, 15 for The Searchers, 112 for Whiplash and 47 for On the Waterfront. These signal fragments for each cut are combined with G.bsanalyze for each audiovisual fragment, obtaining a model signal of the neuronal reaction for the cut for each audiovisual fragment in each electrode. From this process we obtain 31 model signals for each film fragment, one for each electrode, which we will call ASF (Average Signal by Film). The generation of ASF model signals from all users and cuts for each film fragment keeps the common reactions, minimizing the specific ones of the users and minimizing also the different kinds of cuts or the aspects not related to the common event of the shot change. Through this signal processing we have for each electrode four different model signals of reaction to the cut, one for each film fragment, thus being able to compare them and discard what is not common to the cut, such as narrative, aesthetic or technical characteristics of each film fragment. In Fig. 3 we can see graphically the scheme of the creation of ASF signals from the model signals obtained for each audiovisual clip from the electroencephalographic records obtained from the 21 users.

Fig. 3.
figure 3

Scheme of obtaining ASF signals.

Full size image

The ASF model signals obtained are transformed to the frequency domain. For this transform we take sequences of 16 successive samples for each signal in the EEG register timeline (taken at 256 Hz sampling) with the toolbox G.bsanalyze. In this way we obtain the power change for each time interval of 62.5 ms in the frequency domain from the fast Fourier transform. This value called power change represents the oscillation frequency in the EEG register, allowing us to analyse the modulation of neuronal rhythms, corresponding to the electrode and frequency range analysed in response to the experiment stimulus [13, 14]. In our specific case we can know the excitation and inhibition of neurons in different neuronal areas in response to the shot change by cut. Power change values are collected in 4 matrices of dimension 31 × 32, one matrix for each ASF. In each matrix the rows correspond to the electrodes and the columns to the temporal progression representing 62.5 ms intervals.

The power change analysis allows to determine the event-related desynchronization/synchronization. To analyse the responses of each signal in the power change we define the frequency bands in: 0.5–3 Hz for Delta, 3–7 Hz for Theta, 7–14 Hz for Alpha, 14–32 Hz for Beta and 32–42 Hz for Gamma. In addition to this general division, we also perform another segmentation differentiating High and Low ranges of each frequency band. Thus we establish the ranges 0.5–1.5 Hz for Low Delta, 1.5–3 Hz for High Delta, 3–5 Hz for Low Theta, 5–7 Hz for High Theta, 7–10.5 Hz for Low Alpha, 10.5–14 Hz for High Alpha 14–23 Hz for Low Beta, 23–32 Hz for High Beta, 32–37 Hz for Low Gamma and 37–42 Hz for High Gamma. We can see a scheme in the Fig. 4.

Fig. 4.
figure 4

ASF breakdown by frequency ranges.

Full size image

From this process we obtain a model signal of reaction to the cut for each electrode in each of the 15 frequency bands indicated from each ASF corresponding to a film fragment. We get 15 matrices for each of the 4 film fragments, corresponding to the power change values in a specific frequency band, resulting in a total of 60 matrices. Each matrix has 31 rows and 32 columns. Each row being the Power Change values for each electrode in a given frequency band along the selected two seconds of time. The first 16 columns correspond to the values of the second before the cut and the next 16 columns corresponds to the signal after the shot change.

These power change matrices are represented in Eq. 1, where \( M_{i}^{j} \) represents the matrix for the frequency band j and the ASF i. The variable j can take values from 1 to 15, one for each frequency range, and the variable i take values from 1 to 4, one for each audiovisual fragment. Matrix values are represented in Eq. 1 as Electrodex Timey representing the power change value for each electrode along the time. The subscript of each electrode (Ex) is between 1 and 31, while the subscript of time (Ty) is between 1 and 32.

$$ M_{{ASF\left( {i\left[ {1,4} \right]} \right)}}^{{Banda\;de\;frecuencia\left( {j\left[ {1,15} \right]} \right)}} \left( {\begin{array}{*{20}c} {Eectrodo_{1} Tiempo_{1} } & \cdots & {E_{1} T_{32} } \\ \vdots & \ddots & \vdots \\ {E_{31} T_{1} } & \cdots & {E_{32} T_{32} } \\ \end{array} } \right) $$
(1)

4.2 Treatment of the Data Obtained in the ASFs

The next phase of the methodology consists on comparing the data obtained in the ASF. The comparative processes applied on the Power Change of the ASF in the different frequency bands serve to locate those neuronal reactions that have been triggered due to the cut event. In the following two sections we first explain the design of the ASF signal analysis, and then describe its application. Here we define a methodological procedure to locate neuronal reactions due to the event of the shot change by cut.

ASF Signal Analysis Design.

As we designed the experiment with real film fragments, extracted from feature films destined to the general public, in addition to the shot changes by cut there are a great diversity of uncontrolled stimuli, which complicates the identification and isolation of the neuronal reactions due to the cut. To determine which responses are specifically associated with the shot change cut, which is the common event in all the studied signals, we analyse the correlations between the power change for each possible combination of a specific electrode between the four ASF, during the same period of time and in the same frequency band. Thus, to locate in the second after the cut only those reactions associated with the common event of the cut, we search those time intervals where correlations happened between all possible combinations by pairs between the four ASF signals for the same electrode in the same frequency band. This methodology follows the idea proposed by Lachaux, Chavez and Lutz [12], with the difference that their investigation seeks to locate the correlation between different frequency bands in the same stimulus, while we are interested in seeking correlation in the same frequency band, isolating the reaction to the studied stimulus.

It has become customary in neurocinematic studies to compare between the signals recorded by the same electrode in the same frequency band for similar events in order to analyse the neuronal response in the ERD/ERS [2, 5], mainly using ANOVA (analysis of variance). The difference in our approach is that we want to locate the similarities between signals, so we resort to comparison methodologies that detect dependency structures and correlations between the samples instead of resorting to analysis of variance techniques. Another variation that we add in our approach with respect to the referred works is that we want to obtain results in temporal continuity, so instead of establishing watertight periods of time we preformed all analyses with sliding temporal windows.

This dependence and correlation between different ASF signals is the necessary condition to assume that an electrode in a specific frequency band represents a neuronal response due to the cut event, but at the same time, it’s not sufficient to determine the neuronal activity as significant. Then, it is necessary that in addition to the dependence and correlation between the signals of the different ASF, each one contains a significant variation in the power change evolution, indicating that there is a synchronization or desynchronization in the neuronal rhythms (ERD/ERS). This second process discards those dependencies and correlations between signals that are not associated with a significant variation of the power change representing little variation in the neuronal rhythms of the EEG register. The result finally obtained is the location of the temporal windows, the electrodes and the frequency bands that allow us to locate the significant neuronal responses triggered by the event of the shot change by cut.

Procedures for the Search of Neuronal Reactions Due to the Cut Event.

Our methodological approach is based on comparing the power change samples between the four ASF through sliding time intervals for 31 electrodes in each one of the 15 frequency bands. In order to apply it, we establish a work routine using MatLab, which allows us to develop an automated analysis system. To develop the comparison processes between peers of signals, we assign a binary value based on the degree of dependence and correlation for each comparison between the different ASFs for the same electrode in a certain frequency band in a time interval. In this way we assign the value of 1 when a comparison overcome a certain representative threshold value. Comparisons between peers of signals imply a test to analyse the adjustment of depending structures using the permutation test (p-value) and to analyse the correlation between samples through the Spearman test (Rho). These peer comparison tests are applied at time intervals as slide temporal windows. This approach allows us to obtain results with temporal continuity that collect the neuronal reactions to the shot change in the different electrodes and frequency bands.

The comparisons test is made between sliding windows with a range of 6 consecutive power change values. This range is equivalent to 375 ms, because each sample represent an interval of 31.25 ms. The choice of 6 samples to conform the temporal windows responds to the use of the permutations test, due to the fact that if the intervals have less than 5 samples the results can lose robustness [13], and if we have more sampling we would obtain less temporal resolution. Consequently, we segmented the 2 s around the cut from the registration of each ASF (2000 ms) in 27 temporal windows of 6 correlative samples of each temporal window. In this distribution the window consisting of the three temporal samples immediately before and after the cut have the time value 0 and represent the cut.

Equation 2 shows the transformation of the matrix containing the power change values into matrices that contain sliding window vectors. Each element of the resulting matrix, instead of assuming a value as we had before, contain a vector of 6 values. Each vector consists of 6 samples from the power change corresponding to each sliding window. We call it vEkVTz to indicate that it is a vector (v), where the row position indicates the electrode (Ek) and the column position the corresponding time window (VTz). The resulting matrix continues having 31 rows, one for each electrode, but have 27 columns, one for each time window. In consequence, the 14th temporal window (vEkVT14) corresponds to the instant of the shot change by cut, because it is formed by the three samples before the cut (EkT13, EkT14, EkT15) and the three after (EkT16, EkT17, EkT18). Resulting for example vE1VT1 = (E1T1, E1T2, E1T3, E1T4, E1T5, E1T6) and vE31VT27 = (E31T27, E31T28, E31T29, E31T30, E31T31, E31T32).

$$ M_{{i\left[ {1,4} \right]}}^{{j\left[ {1,15} \right]}} \left( {\begin{array}{*{20}c} {E_{1} T_{1} } & \cdots & {E_{1} T_{32} } \\ \vdots & \ddots & \vdots \\ {E_{31} T_{1} } & \cdots & {E_{31} T_{32} } \\ \end{array} } \right) \to M_{{i\left[ {1,4} \right]}}^{{j\left[ {1,15} \right]}} \left( {\begin{array}{*{20}c} {vE_{1} VT_{1} } & \cdots & {vE_{1} VT_{27} } \\ \vdots & \ddots & \vdots \\ {vE_{31} VT_{1} } & \cdots & {vE_{31} VT_{27} } \\ \end{array} } \right) $$
(2)

The identification of dependencies and correlations between signal pairs consists of two phases. The first phase is to apply the permutation to locate these neuronal reactions that show a dependency structure between signals. The second phase consists in using Spearman’s test to know the level and type of correlation between the two signals. By applying both processes we seek to be restrictive in the detection of temporal windows that we can consider as neuronal reactions to the shot change by cut, since the whole process proposed is aimed at isolating the neuronal responses of the cut event from a random set of stimuli derived from the reception of the film fragments showed.

To put it into practice, the analysis applied to each pair of time windows is performed in the two phases described. In the first phase we take two ASFs in a specific frequency band and compare for the same electrode the same time window formed by 6 power change samples. We compare this pair of signals through the permutation test. Setting the threshold of significance at 0.05 [17] we indicate with the value 1 when the result shows a significant dependent structure (p < 0.05). Equation 3 shows how this process operates. The permutations test (PT) is applied to the electrode k for the power change samples included in the time window z (VTz) in the frequency band j (∀ j ∈ [1, 15]). This comparison is made between two different ASFs, so that i and i′ have different values between 1 and 4 (∀ i, i′ ∈ [1, 4]: i ≠ i′). With 4 ASFs there are 6 possible pairwise comparisons between i and i′.

(3)

If the result obtained shows a significant p-value (p < 0.05) it is considered acceptable and is indicated as 1 in a corresponding matrix position where the results are indicated. This technique is an application of the image segmentation methodology used in tomography [18], but we applied it to the segmentation of the range of values resulting from the application of the peer permutation test between 0 or 1, setting the p-value as threshold. The resulting matrix contains all the comparisons made by the permutation test for the same frequency band between two ASFs. Each row corresponds to an electrode and each column to a temporal window.

In this way we detect the temporal windows where the compared data have a significance higher than the p-value threshold, revealing a dependency structure between the compared samples. As a result, we obtain 6 matrices for each frequency band, each one containing the comparisons between two of the 4 ASFs. In the case that in these 6 matrices for a same frequency band, in a time interval of the same electrode it shows a significant dependence, it is considered that this time interval in that electrode and frequency band is reflecting a reaction that could be identified as a neural reaction due to the stimulus of the shot change by cut. In this way, we obtain as a result a final matrix for each frequency band where it is indicated in which electrodes during the same time window occurs a significant dependence relationship in all possible comparisons between the 4 ASFs. The process description is shown in Eq. 4. When in a temporal window and electrode (EkVTz) in the same frequency band (j) occurs that the sum of the pairwise comparisons Boolean results of the 4 ASF matrices i e i’ (∀ i, i′ ∈ [1, 4]: i ≠ i′) results in the value of 6, it means that all the ASFs shows a dependency relationship in that time window and electrode for the defined frequency band. Therefor we assign the value 1 in the corresponding position of a new matrix called PTTotalj, representing j the specific frequency range.

(4)

Applying this procedure to all temporal windows we obtain those that for each electrode and frequency band show dependence between the different film fragments, where the only common event between the 4 ASFs is the shot change by cut. The resulting matrix from this process have a dimension of 31 × 27. The rows correspond to the electrodes and the columns correspond to the temporal windows. We obtain a total of 15 Boolean matrices, one for each frequency band (j).

Once this stage of the process is finished, in the second phase we proceed to determine the goodness of the correlations in the temporal windows detected as dependent. To do this, we apply the Spearman correlation test in the temporal windows where a dependency relationship has been detected by the permutation test. The objective is to verify that the peer relationships established between the same temporal windows of the same electrode in a given frequency band, in addition to showing a dependency relationship, are also correlated. To process it in MatLab, the double condition that the permutations test and the Spearman test is met is easier to obtain separately PTTotal j (Eq. 4) and an equivalent Boolean matrix by applying the Spearman correlation test and compared them.

The Spearman test is applied similarly to the permutations test, establishing pairwise comparisons for each time window of each electrode between the different ASFs in the same frequency range. If the correlation value Rho exceeds the threshold value, it is indicated as 1 in the resulting matrix. To define the correlation between the samples we set the Rho threshold value at 0.5 as stated in abundant literature [19, 20]. The type of correlation that we are looking for among ASFs is positive, since what we are looking for is to detect similar and non-opposite responses. We discard all the Rho negative values because it represents inverse correlations, remaining exclusively the Rho values greater than 0.5 and not those under −0.5, using the Spearman correlation test in one tail form [21]. Equation 5 represents the explained process. For the compared pairs of temporal windows (VTz) of two ASF (between i and i′ being i different from i′) on the same electrode (Ek) in a certain frequency band (j). In case of obtaining a Rho value greater than 0.5 means a positive and significant correlation, then we indicate the result as 1 in the matrix \( SMR_{{i,i^{{\prime }} }}^{j} \) in the corresponding matrix position (EkVTz).

(5)

As we did with the permutation test, we identified as positive only the correlated reactions that exceed the Rho threshold in the 6 possible comparisons between the 4 ASFs, discarding weak correlations. In Eq. 6 we describe this process. When the sum of the results obtained in the same temporal windows for the same electrode (EkVTz) between the different Spearman correlation matrices for each pair of comparison (∀ i, i′ ∊ [1, 4]: i ≠ i′) of the same frequency band (j) results in a value of 6, means that among all the ASF is a correlation at that moment, in this electrode for that frequency band, so we assign the value 1 in the corresponding position of the resulting matrix SMRTotal j.

(6)

The matrix SMRTotal j it allows to identify the temporal windows in which an electrode in a specific frequency band shows correlation in the power change between all the film fragments used. From the whole process we obtain a total of 15 matrices of dimension 31 × 27, where each matrix corresponds to a specific frequency band. In these matrices the rows correspond to the 31 electrodes and the columns to the 27 temporal windows. Thus, we take into consideration only the temporal windows for each electrode in each frequency band where there is a representative dependence relationship and a correlation representative in the recorded signal, reflecting the common event of the cut. We accept these results as evidence of neuronal reactions in the power change due to the shot change by cut, which is the only common event in the 4 film fragments showed.

Next, in order to compare the permutations test and the Spearman correlation test, as both matrix are Boolean, we proceed to multiply the values that occupy the same matrix positions between the matrices PTTotal j y SMRTotal j in the same frequency band (j). In this way, we obtain a Boolean result matrix that meets the double condition of dependence and correlation, indicated as 1 only the temporal windows where p-value is less than 0.05 and Rho greater than 0.5 in all possible combinations between entre i and i’ (∀ i ≠ i′). In Eq. 7 we indicate the multiplication operation between the values that occupy the same matrix position as “.x”, similar to the execution command in MatLab (.*), to differentiate the notation used from a standard matrix multiplication. The result of this operation is the location of the temporal windows where there is correlation and dependence between the 4 ASFs for the same electrode in the same frequency band (j). That is, the value 1 identifies in the resulting matrix all those temporal windows where a neuronal reaction due to the event of the cut is detected. We call this final matrix CD (Cut Dependencies).

$$ CD_{31x27}^{j} = PTTotal_{31x27}^{j} \;.x\;SMRTotal_{31x27}^{j} $$
(7)

The resulting matrix CD j has a dimension 31 × 27, where each row represents an electrode and each column a temporal window. 15 matrices are obtained, each for an analysed frequency band (j). In these matrices, the temporal windows of each electrode that show a neuronal response due to the cut event contain the value 1, while the rest contain the value 0.

The data obtained from the proposed analysis allows a temporal study of the neuronal response triggered by the cut event for each frequency band and electrode. Therefore, the last step in the methodology is to identify those temporal windows that, in addition to showing reactions triggered by the change of plane by section, contain significant variations in the temporal evolution of the power change. For this we resort to the analysis of the slopes of the signals [22, 23]. This process is performed by calculating the slopes separately in the different time windows, after their transformation into a Boolean matrix and finally multiplying the values that occupy the same position of the matrix obtained with the also Boolean matrix CD, already calculated above. In this way, as we operate with Boolean matrices, we get 1 when the temporal windows detected as dependent on the cut event contain a slope that exceeds the threshold and maintains the same sign in the 4 ASF.

Variations in the evolution of power change indicate a process of synchronization or desynchronization in neuronal rhythms [24, 25]. Therefore, if we detect significant slopes in the temporal windows from an electrode at a frequency band in which we have detected a possible neuronal response due to the shot change by cut, we obtain as a result the location of important synchronization or desynchronization processes as consequence of the cut. Then, for all temporal windows of each electrode in each frequency band where dependence and correlation has been detected in the matrix CD j(EkVTz) we analyse the slopes in the power change values of the 4 ASFs.

To perform this process, we take each vector formed by the 6 temporal samples corresponding to the time window (vEkVTz) in each of the 4 ASFs and we calculate their slope. If the slopes calculated in the temporal windows exceed a representative threshold in the 4 ASFs with the same sign, the result is marked as 1, indicating all others as 0. It is important to consider the sign of the slope, because we can have positive (ERS) or negative (ERD) slopes, and only results that show the same type of variation in neuronal rhythms for the 4 ASFs in the same time window of the same electrode and in the same frequency band can be representative. Therefore, we consider as relevant in terms of variation of the power change only those slopes that exceed a certain threshold with the same sign in the same time window, electrode and frequency band for the 4 ASFs.

To establish a threshold value that defines the level of variation to due to the shot change by cut as representative, we evaluate the slope of the power change. This evaluation must be weighted in relation to a state of rest or balance. To establish the threshold value of the slope that identifies such variations of the power change as representative, we analyse the data contained in the second before the shot change (baseline) as a reference. When performing operations with sliding windows of 6 samples from the power change, we obtain 27 temporal windows in the 2 s analysed, one second previous to the cut and one second after. From these temporal windows we consider the intermediate one as the instant of the cut, because it is composed by 3 samples before and three after the shot change. This means that the two temporal windows prior to the intermediate one will also contain samples after the cut (Fig. 5), so if we consider them as baseline, they can contaminate the reference previous to the stimulus with neuronal reactions after the shot change. Therefore, in order to establish the threshold value that determines the representative slopes in the temporal windows of the power change, we use the first 11 temporal samples as a baseline, excluding the temporal windows 12 and 13, as we can see in the Fig. 5.

Fig. 5.
figure 5

Temporal sliding windows that are not influenced by power change samples after the shot change by cut.

Full size image

In neurological studies, the segmentation of data in “below average” and “above average” is usually used in order to discriminate between two groups depending on a specific characteristic that is considered to be more represented in one of the two parts than what would be the case in the whole data set [26,27,28]. In our case, we take as significant slopes in the temporal windows of the baseline those above average. Therefore, we set the threshold value as the average obtained in the defined baseline. To calculate the average, we analyse the slopes in the sliding windows of the baseline formed by the first 11 sliding windows that do not contain samples after the shot change by cut. From the baseline sliding windows, we select as representative those that show a common behaviour, eliminating all those windows that for the same electrode, in the same frequency band and in the same time window do not have the same sign in the 4 ASF. We only consider those cases in which the slope has the same increasing or decreasing trend in the 4 ASF. We carry out this previous operation to obtain the average data in a similar way to how the detection of the significant slopes on the temporal windows will be worked after the cut. We are interested in knowing the average inclination in degrees to calculate the deviation either positive or negative with respect to 0°, so we need to know the average inclination at its absolute value. For this we calculate the average slope from the absolute values of each slope and thus finally obtain the threshold value that determines whether a slope value is significant.

Once the threshold of significance for the slopes is established, we proceed to operate in the different temporal windows in order to know the inclination of the neuronal reactions due to the event of the cut. To do this we begin by calculating the slopes of each time window, as defined in Eq. 8. As we calculate the slope in 6 sample windows, the denominator is 6 and the samples at the ends of each time window are Tz y Tz + 5 for each frequency band (j) in each ASF (i).

$$ Slope_{i}^{j} \left( {vE_{k} VT_{z} } \right) = actan\left( {\frac{{E_{k} T_{z + 5} - E_{k} T_{z} }}{6}} \right) $$
(8)

We create a matrix \( Sl_{i}^{ j} \left( {E_{k} VT_{z} } \right) \), showed in Eq. 9, from the matrix \( Slope_{i}^{ j} \left( {vE_{k} VT_{z} } \right) \), which contains the values of the slopes for each specific frequency band (j) in a specific ASF (i) for each electrode (Ek) and each time window (VTz), to calculate the slope from sliding windows of 6 temporal samples on each electrode (vEkVTz). Once the slopes are calculated, if they exceed the positive threshold they are indicated in the matrix Sl as 1, and if they exceed the negative threshold they are indicated as −1. The rest of the matrix positions are indicated as 0. This differentiation between 1 and −1 will allow us to ensure that the slopes of each time window have the same slope sign in the 4 cases when comparing within the different ASFs.

(9)

In this way we obtain, as indicated in Eq. 9, the identification of all the slopes that exceed the threshold in a time window. This process results in 15 matrices for each film fragment (i), each one corresponding to its frequency band (j). The matrices Sl have a dimension 31 × 27, the rows corresponding to the electrodes and the columns corresponding to the temporal windows.

When applying the slope analysis, it is important to ensure that we locate those whose variation in the power change represent a monotonous behaviour, that is, whose slope calculated from the first and last value of the time window of 6 temporal samples is consistent with the trend of the slope throughout the interval. For this, we verify that within each window of 6 temporal samples where a representative slope is located, that window also contains in its samples at least an interval with a slope of 3 samples that is also representative and have the same sign. If we detect that the slopes are not monotonous we change the value 1 for 0 in the corresponding matrix position.

Once the power change slopes are known for each time window, we proceed to compare each one with their equivalent in the same electrode and frequency band for the different ASFs. In Eq. 10 we can see how the matrix SlopeTotal j referring to the significance of the slopes in the power change between the different ASFs is obtained for each frequency band j, which later we multiply, position by position, by the corresponding matrix CD j previously obtained (Eq. 7). This procedure allows us to locate the temporal windows that show a correlation and dependence between the neuronal reactions triggered by the shot change by cut, which have an important variation in the power change, reflecting a variation in the neuronal rhythms that we will analyse through the ERD/ERS.

The matrix SlopeTotal j is obtained through the sum of each Boolean value assigned to the calculation of the slope made on the power change in the 4 ASFs (i) in windows composed of 6 temporal samples (VTz) for the same electrode (Ek) in the same frequency band (j). As previously explained, each slope of a temporal window that exceeds the defined threshold is indicated as +1 or −1. To transform the matrix SlopeTotal to Boolean and be able to operate it with the matrix CD, we take only as acceptable the temporal windows in which all the slopes for the same frequency band in the same electrode for the 4 ASFs have an inclination that exceeds the threshold, so that the optimal results of the summation should be 4 or −4. Each matrix position where the result of the sum is 4 or −4 is replaced by 1. All the other positions are indicated as 0 because they are not relevant. The process is explained in the Eq. 10.

(10)

In Eq. 10, we obtain a Boolean matrix (SlopeTotal j) of dimension 31 × 27 that identifies whether there is an instance in a specific time period of the power change that exceeds a certain threshold with the same sign on the same electrode for the 4 ASFs in the same frequency band (j). The rows of the matrix SlopeTotal correspond to the 31 electrodes and the columns to the 27 temporal windows.

Once the matrix SlopeTotal j is obtained for each frequency band j, we proceed to combine it with the matrix CD j previously obtained. In this way we locate those temporal windows that have a synchronization or desynchronization process due to the event of the cut. The process is explained in Eq. 11. To combine these matrices, we multiply the values that occupy the same matrix positions (.x) between SlopeTotal j and CD j for the same frequency range (j). Through this process we obtain the matrix DSC j (Desynchronizations and Synchronizations due to the Cut) for each frequency band (j).

$$ DSC_{31x27}^{j} = CD_{31x27}^{j} \;.x\;SlopeTotal_{31x27}^{j} $$
(11)

The matrix DSC obtained in Eq. 11 is a Boolean matrix where each time window is indicated as 1 if a neuronal reaction triggered by the cut event has been detected and shows a significant process of synchronization or desynchronization. We obtain 15 matrices DSC, one for each frequency band. Each matrix DSC has a dimension 31 × 27, where the rows correspond to the electrodes and the columns to the temporal windows.

When combining the results of the monotonous slopes with the results obtained by permutation test and correlations of Spearman, all made with sliding windows of 6 samples, we are sure that the representative slopes that occur in the 4 ASFs for the same time in the same electrode and frequency band are related. Therefore, we performed a security check as similar as possible, whether for the analysis of the slopes in the windows of the three temporal samples or for the six temporal samples. We cannot reliably apply the permutation test on three-sample windows [16], but we can apply the Spearman correlation test and get acceptable results [29]. In this way, in addition to verifying that the behaviour of the slope is monotonous, correlated and dependent on the slope for the temporal windows of 6 samples, we also verify that the relationship between the 4 ASFs is maintained despite reducing the size of the sliding window. That is, when we detect dependence, correlation and a significant slope in a period of 6 samples among the 4 ASFs, we ensure that a segment of 3 samples contained in this period also shows correlation and a significant slope.

In summary, we consider that a specific temporal window of an electrode in a given frequency band reacts significantly due to the shot change by cut, either in the form of synchronization or desynchronization, when three established conditions are met: overcome the p-value in the permutation test, overcome the Rho in the Spearman’s correlation and have a monotonous slope that overcome the defined threshold. All the mathematical processing to get the results is carried out automatically by means of scripts executed by MatLab. Thanks to programming these scripts we can perform a depth analysis that automatically locates the significant reactions in the power change due to the cut event.

4.3 ERD/ERS Study of Temporal Windows Identified as Dependents to the Cut

Once the neuronal reaction triggered by the cut event are located knowing each temporal window for electrode and frequency band, we proceed to analyse its ERD/ERS from the registered power change. The analysis of the ERD/ERS is based on the detection of variations in neuronal activity in relation to the state of non-reaction to the event (baseline) to be able to make an observation in terms relative to a state prior to the variation of the neuronal rhythms. This variation is calculated in percentage terms with respect to a state of no reaction to the analysed event.

To quantify the variation of the power change that occurs in an electrode in a certain frequency band we do it with respect to the average value of the signal in the second prior to the shot change by cut (baseline) [9]. To obtain the value of the ERD/ERS we apply Eq. 12 [24, 25]:

$$ ERD = \frac{Baseline - Test}{Baseline} \times 100 $$
(12)

Applying Eq. 12 we obtain the ERD/ERS values as a percentage. Resulting from this operation, positive values correspond to desynchronization processes (ERD) and negative values to synchronization process (ERS). This system allows us to understand the evolution of the neuronal rhythms in the spectator facing the shot change by cut.

5 Discussion

The application of the methodological design though automated scripts in MatLab allowed us an intensive and complete analysis for all the electrodes in all the frequency ranges over slide time windows, allowing us to locate and identify any variation that affects Power Change as a consequence of the event of the cut. This allows us to be able to perform an in-depth analysis for all the data registered through the EEG, without having to limit the analysis to specific data biases and also avoids the need to produce ad hoc audiovisual material for the laboratory [2]. Until now, for the analysis of the spectators by means of the EEG, audiovisual clips have been produced specifically for laboratory conditions in order to assess specific cinematographic technical aspects [2, 8, 9]. Fragments extracted from real films have been used only to study emotions where specific technical aspects of the film were not taken into account [30, 31]. The methodology developed in the present investigation allows us to obtain successful results concentrating on cinematographic techniques from films fragment originally designed for the consumption of cinematographic spectators, allowing to make a contribution to the cinematographic theory based on the analysis of the film itself and not from audiovisual recreations filmed for laboratory purposes.

In the methodology two processes are established. A first one oriented to detect all the temporal windows for each electrode and frequency band where neuronal reactions are a consequence to the shot change by cut, and a second process where we identify which of these neuronal reactions constitute representative variations in the neuronal rhythms. Following the exposed methodology we can locate the event of the shot change by cut recognizing modulation patterns in the frequency bands Theta and Delta. We can identify activity in different cortical areas the first 250 ms after the cut, most of which are Theta synchronization. In addition, we can locate a clear theta synchronization tendency between 700 ms and 1000 ms, which coincides with a desynchronization of Delta in the same period. Most of the activity found is localized in the parietal area. The results obtained indicate a clear involvement of the hippocampus in the event of the shot change by cut, which supports our methodology because it coincides with the results of previous investigations through fMRI [1].

6 Conclusion

The present study proposes a novel methodological approach for the detection of neuronal responses triggered by the shot change by cut. The use of real film fragments that are stylistically diverse for the study of the EEG on aspects related to the cinematographic technique requires debugging all those reactions due to the film context, allowing to be able to analyse films designed for the spectator and not audiovisual clips filmed for the experiment itself. The exposed methodology for the detection of temporal windows in electrodes and frequency bands in which neuronal reactions registered in the power change occur due to the stimulus of the cut, although it is very restrictive, it allows us to extract clear results specifically associated with the perception of the shot change by cut which, at the same time, are backed by previous investigations [1]. The restrictiveness of the designed system means that we only consider 1.02% of temporal windows, which are those dependent and correlated between the 4 ASFs, and 0.02% which are dependent and correlated with significant variations in their Power Change between the 4 ASF.

To adapt Brain-computer interfaces to real world situations we need work with noise situations. Stimuli from audiovisual clips are complex as real life ones. Previous experiments works under the noise absence’s premise. Therefore, methodologies involved in events detection thru EEG analysis need to transcend simplicity of those experiments. Our approach shows how it is possible to isolate a particular kind of event using the proposed framework. To find brain response, which a quick EEG analysis can process almost in real time, it is necessary to previously identify the patterns associated with the events. We show here a working framework adaptable to many events detection that need to identify time lapses and brain regions involved in human response to such events.