Attention with kernels for EEG-based emotion classification

Abstract

A kernel attention module (KAM) is presented for the task of EEG-based emotion classification using neural network based models. In this study, it is shown that the KAM method can lead to more efficient and accurate models using only a single parameter design. This additional parameter can be leveraged as an interpretable scalar quantity for examining the overall amount of attention needed during deep feature refinement. Extensive experiments are analyzed on both the SEED and DEAP datasets to demonstrate the module’s performance on subject-dependent classification tasks. From these benchmark studies, it is shown that KAM is able to boost the backbone model’s mean prediction accuracy by more than 3% on some subjects and up to more than 1%, on average, across 15 subjects in the SEED dataset for subject dependent tasks. In the DEAP dataset, the improvement is more significant by achieving greater than 3% improvement in the overall mean accuracy versus the no-attention case, and more than 1–2% when benchmarked against various other state-of-the-art attention modules. In addition, the predictive dependencies of KAM with respect to its single parameter is numerically examined up to first order. Accompanying analyzes and visualization techniques are also proposed for interpreting the KAM attention module’s effects, and interaction with the backbone model’s predictive behaviors. These quantitative results can be explored in greater depth to identify correlations with pertinent clinical neuroscientific observations. Finally, a formal mathematical proof of KAM’s permutation equivariance property is included.

Appendix

Permutation matrix A permutation can be represented as a matrix P whose elements only take values in $\{0, 1\}$. Note that the sum of each row (column) from a permutation matrix P is always 1. Also note that $\text {PX}$, by definition, will reorder X’s rows, and XP will reorder X’s columns. As an example, consider the matrix/vector system:

$$\begin{aligned} P = \begin{bmatrix} 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} 1 \\ 1 &{} 0 &{} 0 \end{bmatrix}, X = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} \quad \text {such that}\quad \text {PX} = \begin{bmatrix} 2 \\ 3 \\ 1 \end{bmatrix}. \end{aligned}$$

It is immediately clear in this example that if $P_{\text {ij}} = 1$, P will send X’s j-th row to the i-th row by left multiplication. Moreover, P is unitary: $P^TP = P^TP = I$.

Lemma 1

The kernel matrix $M_K(X; \theta )$ has the property:

$$\begin{aligned} M_K(PX;\theta ) = P M_K(X;\theta ) P^T, \end{aligned}$$

(2)

for any permutation matrix P.

Proof

Suppose the feature block X has shape $m\times n$. A permutation matrix P ($m\times m$) sends the u-th row of X to the i-th row, and the v-th row of X to the j-th row, i.e., $[PX]_i = X_u$ and $[\text {PX}]_j = X_v$. In terms of P this implies that $P_{\text {iu}} = 1, P_{\text {is}} = 0$ for $s \ne u$ and $P_{\text {jv}} = 1, P_{\text {jt}} = 0$ for $t\ne v$.

On the left side, the matrix value at location (i, j) then becomes:

$$\begin{aligned} M_K(\text {PX};\theta )_{\text {ij}}&= \exp (-\alpha \text {d}([\text {PX}]_{i}, [\text {PX}]_{j})^2 )&\\&= \exp (-\alpha \text {d}(X_{u}, X_{v})^2 )&\\&= M_K(X;\theta )_{\text {uv}}, \end{aligned}$$

while on the right side,

$$\begin{aligned} {[}\text {PM}_K(X;\theta ) P^T]_{\text {ij}}&= \sum _{s} P_{\text {is}} [M_K(X;\theta ) P^T]_{\text {sj}}&\\&= \sum _{s} P_{\text {is}} \sum _{t} M_K(X;\theta )_{\text {st}} (P^T)_{\text {tj}}&\\&= \sum _{s} \sum _{t} P_{\text {is}} M_K(X;\theta )_{\text {st}} P_{\text {jt}}&\\&= P_{\text {iu}} M_K(X;\theta )_{\text {uv}} P_{\text {jv}}&\\&= M_K(X;\theta )_{\text {uv}}. \end{aligned}$$

It is easy to see from the above proof that the conclusion does not depend on the choice of distance function, e.g., $L^p$ norms for any p can also be used.⁴$\square$

Theorem 1

The proposed KAM attention layer as shown in Eq. (1): $\varvec{\psi }(X):= [I + M_K(X; \theta )]X$ is permutation equivariant. That is, for a permuation matrix P, $\varvec{\psi }(\text {PX}) = P\varvec{\psi }(X)$

Proof

This result follows by a direct calculation from Lemma1, as:

$$\begin{aligned} \varvec{\psi }(\text {PX})&= [I + M_K(\text {PX}; \theta )]\text {PX}&\\&= \text {PX} + \text {PM}_K(X; \theta )P^T\text {PX}&\\&= \text {PX} + \text {PM}_K(X; \theta )X&\\&= P[I + M_K(X; \theta )]X&\\&= P\varvec{\psi }(X). \\ \end{aligned}$$

$\square$