Parallel linear dynamic models can mimic the McGurk effect in clinical populations

Abstract

One of the most common examples of audiovisual speech integration is the McGurk effect. As an example, an auditory syllable /ba/ recorded over incongruent lip movements that produce “ga” typically causes listeners to hear “da”. This report hypothesizes reasons why certain clinical and listeners who are hard of hearing might be more susceptible to visual influence. Conversely, we also examine why other listeners appear less susceptible to the McGurk effect (i.e., they report hearing just the auditory stimulus without being influenced by the visual). Such explanations are accompanied by a mechanistic explanation of integration phenomena including visual inhibition of auditory information, or slower rate of accumulation of inputs. First, simulations of a linear dynamic parallel interactive model were instantiated using inhibition and facilitation to examine potential mechanisms underlying integration. In a second set of simulations, we systematically manipulated the inhibition parameter values to model data obtained from listeners with autism spectrum disorder. In summary, we argue that cross-modal inhibition parameter values explain individual variability in McGurk perceptibility. Nonetheless, different mechanisms should continue to be explored in an effort to better understand current data patterns in the audiovisual integration literature.

Appendix A

1.1 Model specifications

The linear dynamic parallel interactive model and its parameters were derived from the systems utilized by Eidels et al. (2011), and previously by Townsend and Wenger (2004) to describe how interactions can lead to changes in capacity. This Appendix also appears in Altieri (2016). The model is specifically tailored to include three auditory and visual inputs along with the internal representations of these phonemic categories. As an example, the vector x(t) represents the level of accumulation or activation for a given phonemic category (x _b (t) = Auditory /b/, for example):x(t)= $\left[\begin{array}{c}\hfill {x}_b(t)\hfill \\ {}\hfill {x}_g(t)\hfill \\ {}\hfill \vdots \hfill \end{array}\right]$. Rows 3–6 of the vector represent the level of evidence accumulated for auditory /d/, and visual /b/, /g/, and /d/ respectively. Next, the following vector u denotes the level of input to each phonemic category in the auditory (first 3 rows) and visual (last 6 rows) modalities:u= $\left[\begin{array}{c}\hfill {u}_{/b/}\hfill \\ {}\hfill {u}_{/g/}\hfill \\ {}\hfill \vdots \hfill \end{array}\right]$ (One simplifying assumption here is that the phonemic inputs are specified by constants rather than functions (e.g., t > 0:u ₁(t) = u ₁ ; u ₂ (t) = u ₂)).

A complete model with three auditory and three visual inputs with visual to auditory (V➔A) and auditory to visual (A➔V) connection will require a 6 × 6 activation matrix; this can be represented by A = $\left[\begin{array}{ccc}\hfill {a}_{A/b/}\hfill & \hfill \cdots \hfill & \hfill {a}_{A/b/;V/d/}\hfill \\ {}\hfill \vdots \hfill & \hfill \ddots \hfill & \hfill \vdots \hfill \\ {}\hfill {a}_{A/d/;V/b/}\hfill & \hfill \cdots \hfill & \hfill {a}_{V/d/}\hfill \end{array}\right]$. (For simplicity, within modality interactions will be left out of this version of the model). In this model, the diagonal elements represent the stabilizing parameter for each auditory and visual accumulator. For example, a _A/b/ represents the stabilizing parameter for auditory /b/ (i.e., A/b/) and a _V/d/ represents the stabilizing parameter for visual /d/, which can be found in row 6 and column 6. These cross-modal inhibition parameter, particularly V/g/ ➔ A/b/, will be important in the following simulations because the lip-motion used to make a visually articulated /g/ is incompatible with the lip-motion used to make a /b/ sound; as a result, we can use a negative number for this parameter to include visual inhibition in the model.

Assuming separate sources of noise, setting the cross-modal interaction parameters (e.g., a ₁₂ and a ₂₁) to “0” renders the model mathematically equivalent to an independent parallel model with unlimited capacity (see Miller 1982; Townsend and Nozawa 1995). Eidels et al.(2011) utilized parameter values that enforced stability in the system; stability was maintained in the system by setting the stabilizing parameters such as a_A/b/ and the interaction parameters |a_A/b/;V/d/| < |a_A/b/|. (The model can be further simplified by assuming equivalent accumulation rates and cross-channel activation for all of the parameters of interest.). Refer to Townsend and Wenger (2004), for discussion on an early interaction matrix denoted by B (Integration, or cross-modal interaction occurring at these early perceptual stages, will not be discussed).

The differential equation (Eq. (1)) constitutes a deterministic (i.e., we do not yet include Gaussian noise) model with three auditory and three visual phonemic inputs:

$$ \frac{d}{dt}\mathbf{x}(t)=\mathrm{A}\mathbf{x}(t)+\mathbf{u}(t)=\left[\begin{array}{ccc}\hfill {a}_{A/b/}\hfill & \hfill \cdots \hfill & \hfill {a}_{A/b/;V/d/}\hfill \\ {}\hfill \vdots \hfill & \hfill \ddots \hfill & \hfill \vdots \hfill \\ {}\hfill {a}_{A/d/;V/b/}\hfill & \hfill \cdots \hfill & \hfill {a}_{V/d/}\hfill \end{array}\right]\cdot \left[\begin{array}{c}\hfill {x}_b(t)\hfill \\ {}\hfill {x}_g(t)\hfill \\ {}\hfill \vdots \hfill \end{array}\right]+\left[\begin{array}{c}\hfill {u}_{/b/}\hfill \\ {}\hfill {u}_{/g/}\hfill \\ {}\hfill \vdots \hfill \end{array}\right] $$

(1)

Importantly, for the deterministic system, there exists a closed form solution to the differential equation describing the level of activation in each channel at time t. Eq. (2) shows an example interaction in the accumulation rates between auditory /b/ and visual /d/ within a hypothetical listener. The full model showing a graphical view of the interactions is specified in Fig. 1c (A version of this equation was implemented in simulations carried out by Eidels et al. (2011).¹).

$$ \begin{array}{l}{x}_{A/b/}(t)=\frac{u_{A/b/}+{u}_{V/g/}}{2\left({a}_{A/b/}+{a}_{A/b/;V/g/}\right)}\;\left[ \exp \left[\left({a}_{A/b/}+{a}_{A/b/;V/g/}\right)t\right]-1\right]+\frac{u_{A/b/}-{u}_{V/g/}}{2\left({a}_{V/g/}-{a}_{A/b/;V/g/}\right)}\left[ \exp \left[\left({a}_{A/b/}-{a}_{A/b/;V/g/}\right)t\right]-1\right]\\ {}{x}_{V/g/}(t)=\frac{u_{V/g/}+{u}_{A/b/}}{2\left({a}_{V/g/}+{a}_{A/b/;V/g/}\right)}\;\left[ \exp \left[\left({a}_{V/g/}+{a}_{A/b/V/g/}\right)t\right]-1\right]+\frac{u_{V/g/}-{u}_{A/b/}}{2\left({a}_{V/g/}-{a}_{A/b/;V/g/}\right)}\;\left[ \exp \left[\left({a}_{V/g/}-{a}_{A/b/;V/g/}\right)t\right]-1\right]\end{array} $$

(2)

The model was made stochastic by adding independent and identically distributed Gaussian white noise Ƞ(t) to each of the inputs. The stochastic version of the differential equation describing a two-channel model with cross-talk between channels is provided in Eq. (3). The interested reader is referred to Higham (2001), Øksenda (1985), and Smith (2000) for tutorials that include methods for working with and approximating stochastic differential equations.

$$ dx(t)= Ax(t)dt+u+\sigma I+ dB(t) $$

(3)

In Eq. (3) , A is a 6 × 6 feedback matrix, u represents the 6 × 1 input vector, σI represents the 6 × 6 matrix denoting the error, and B ( t ) represents the six-dimensional Brownian motion process.

1.2 Decision process

In the interactive model, perceptual evidence for different categories (words, syllables, or phonemes or visual visemes) accrues in separate channels. Activation (i.e., x ( t )) is accumulated over a finite time period although the model, in the present form, does not yet account for the decision process required for phoneme or “speech recognition”. To accomplish this, the model must be augmented in a couple of key ways: First, a decision threshold is necessary in order to determine when enough phonemic evidence has accumulated in a specific channel (e.g., auditory /b/, auditory /d/, etc.) for a response. This can be accomplished by adding accumulation thresholds to the auditory and visual channels, which can be represented by the parameter “γ”.

How is a decision made? A logical gate, in this case an OR gate, is imposed on the system. It stipulates that recognition occurs at the time point in which the first auditory phoneme, represented by x _i ( t ), reaches the threshold. While visual (viseme) activation is included in the model, the decision only involves auditory phonemes; hence, the race is really only among auditory phonemes. Logical gates thus represent the decisional aspects of the system in the sense that they decide whether processing finishes when one auditory phoneme reaches thresholds (OR), or instead, whether it needs to wait for multiple channels such as each of the three auditory phonemes and/or visual phonemes in this model to finish processing (AND). The termination rule determines the form of the cumulative distribution function (CDF) of RTs. The CDF for the OR rule is given by P(OR RT ≤ t) = P(T_A/b/ ≤ t OR T_A/g/ OR T_A/d/ ≤ t) = P(x_/b/(t) > γ OR x_/g/(t) > γ OR x_/d/(t) > γ). T_/b/, T_/g/, and T_/d/ denote random variables for RTs on the auditory channel. It is possible to slightly amend the model to allow for decisions once an auditory or visual phoneme reaches threshold. In this latter case, if a viseme reached threshold first, then the viewer would recognize the input by sight rather than audition.