Contributed articleA model of the combination of optic flow and extraretinal eye movement signals in primate extrastriate visual cortex: Neural model of self-motion from optic flow and extraretinal cues
Introduction
In higher mammals and humans, the perception of space and of motion through space is largely based on vision. Movements of the eyes in the head massively change the visual input though we continue to perceive a stable world around us. Questions about the use of extraretinal signals for the perception of such spatial stability during eye movements have long been raised and much discussed. Two classical theories present the two opposing approaches to this issue. The `inferential' theory proposes that, in order to attain stability of the visual world during eye movements, an extraretinal signal, e.g. an efference copy of the motor command driving the eye muscles, is used to compensate for the visual effects of the eye movement (von Helmholtz, 1896; Mittelstaedt, 1990; Sperry, 1950: Von Holst and Mittelstaedt, 1950). An opposite point of view is taken by the theory of `direct perception' (Gibson, 1950). In this view, no extraretinal signal is required, and the perception of the movement of objects in the world or of the observer himself is solely mediated from the visual information alone. In recent years, however, it has become evident that both theoretical views were for one reason or another inappropriate, and that aspects from both sides have to be considered (see Wertheim (1994) for a review). One important point in this regard is that the extraretinal signal typically underestimates the speed of the eye movement (Wertheim, 1987; Honda, 1990) and operates with a variable gain (Haarmeier and Thier, 1996).
A special case of this discussion concerns the detection of the direction of heading in the presence of eye movements. Eye movements here refer to slow eye movements such as smooth pursuit or the slow phases of the optokinetic and vestibulo-ocular reflexes. Since the original suggestion by Gibson (1950) that the optic flow field, i.e. the visual motion induced by self-motion in a stable world, provides all necessary information for navigation in an unknown environment, several computational (Bruss and Horn, 1983; Heeger and Jepson, 1992; Hildreth, 1992; Koenderink and van Doorn, 1987; Longuet-Higgins and Prazdny, 1980; Rieger and Lawton, 1985) and psychophysical (Rieger and Toet, 1985; van den Berg, 1992; Warren et al., 1988; Warren and Hannon, 1990) studies have supported this view. In the psychophysical experiments, the performance of human subjects in two conditions were compared. In the first condition, an optic flow stimulus simulating observer translation was presented while the subjects tracked a small pursuit target with his eyes. In the second condition, the subject kept his eyes stationary, while the stimulus simulated the visual effect of the eye movement together with the observer translation. Often there were few differences between the two conditions. There are, however, experimental situations which do require extraretinal signals. Most notably, a movement towards a frontoparallel plane (Regan and Beverly, 1982; Warren and Hannon, 1990) gives rise to a characteristic misjudgment of the direction of heading when only optic flow is available as a cue. However, also for movements on top of a ground plane, extraretinal signals are required, depending on the type of the eye movement. When eye movements simulate the pursuit of a horizontally moving target, humans cannot correctly judge the direction of heading (Royden et al., 1994). When the same eye movements are actually performed, i.e. when extraretinal information is available, the judgments are correct. In contrast, when eye movements simulate the stabilization of gaze on an environmental point, no extraretinal signal is required for a correct perception (van den Berg, 1993). The different simulated eye movements in these two situations give rise to fundamentally different flow patterns on the retina (Lappe and Rauschecker, 1995b). The different results can be linked to this difference in structure of the optic flow and to the similarity of the stimulus to the optic flow experienced during movement on a curved path (Lappe and Rauschecker, 1994; Royden, 1994).
Taken together, the present results from psychophysical studies suggest that the human visual system can make use of the information available in the optic flow field to determine the direction of heading. However, extraretinal input can improve the performance in many cases and is especially required to disambiguate the visual information in certain problematic situations.
The medial superior temporal area, or area MST, is regarded as a cortical site where the determination of egomotion parameters might take place (Duffy and Wurtz, 1991a, Duffy and Wurtz, 1995; Lagae et al., 1994; Pekel et al., 1996). Lappe and Rauschecker proposed a neural network model of heading detection from optic flow that achieves the visual response properties of MST neurons from the output of MT-like neurons (Lappe and Rauschecker, 1993b) and links these properties to the psychophysics of human heading detection from optic flow (Lappe and Rauschecker, 1995a, Lappe and Rauschecker, 1995b). Recent experimental studies (Duffy and Wurtz, 1995; Pekel et al., 1996; Lappe et al., 1996) have confirmed the predictions from this model.
However, besides its specificity for optic flow analysis, area MST is also linked to eye movements. `Pursuit' or `visual tracking' neurons in MST, discharge during ongoing smooth pursuit (Erickson and Dow, 1989; Komatsu and Wurtz, 1988) and receive extraretinal signals about the occurrence of a pursuit eye movement (Newsome et al., 1988; Ilg and Thier, 1996b). While area MST is involved in the generation of smooth pursuit (Dürsteler and Wurtz, 1988; Komatsu and Wurtz, 1988), these neurons might also provide area MST with a signal about the presence of an eye movement (Erickson and Thier, 1991). As with the extraretinal input to the pursuit neurons (Newsome et al., 1988) this distinction is not found earlier in the visual motion pathway, neither in area MT (Erickson and Thier, 1991), nor in area V1 (Ilg and Thier, 1996a).
This article presents a model of heading detection that includes not only the processing of visual, but also of extraretinal signals. In order to account for the psychophysically observed capabilities of the human visual system in the task of heading detection, neither of the two classical theories outlined above, the `inferential' or the `direct' theory, is sufficient. Instead, the psychophysical results suggest a situation-dependent combination of visual and extraretinal signals. Royden (1994) has proposed that the visual system uses an extraretinal signal to remove the eye-movement-induced motion from the retinal flow field and subsequently estimate translation and rotation on a curved path. The model presented here follows a similar approach, since it also first removes the eye-movement induced visual motion and then analyses the resulting `purified' flow field. However, different from Royden (1994) the subsequent analysis of the purified flow field still allows for residual influences of eye movements that might result from imperfect compensation. Moreover, the model provides a neuronal basis for this proposed combination of visual and extraretinal signals. Also, the paper will not be concerned with motion on a curved path, but rather with self-translations along a straight line.
The proposed model assumes that an extraretinal signal is first used to estimate the eye movement induced retinal image motion and then compensate for it. Consistent with experimental evidence (Haarmeier and Thier, 1996; Honda, 1990; Wertheim, 1987) this compensation mechanism must be assumed to be incomplete since it accounts only for a certain proportion of eye speed. Therefore the resulting purified flow field is nevertheless processed by a heading detection mechanism that allows for the presence of eye rotation and takes visual evidence of eye movements into account. The visual heading detection is based on an earlier developed model (Lappe and Rauschecker, 1993b). This earlier model does not include any extraretinal input but retrieves heading directions visually from retinal flow even in the presence of eye movements. It is successful in many situations and consistent with human psychophysical data in a way that the earlier model fails in the case of approaching a frontoparallel plane, in which extraretinal input is required. This paper describes how extraretinal input can be combined with this model such as to also retrieve heading in this more complicated situation. The interaction of visual and extraretinal signals in the model is based on known properties of MST neurons. The combination of visual and extraretinal signals is performed by optic flow processing MST-like model neurons that receive input from direction-selective MT-like model neurons and from MST-like model pursuit neurons. The basic mechanisms of this interaction, i.e. the requirements for the appropriate synaptic connections, have been briefly presented earlier (Lappe et al., 1994). This article will, after presenting the combined model in Section 2, compare it to psychophysical and neurophysiological data (3 Extraretinal input disambiguates flow field information, 4 Consequences of extraretinal input for single neurons in the model of area MST), and discuss the results and implications (Section 5). The comparison to psychophysical findings will focus on the case of an observer approaching a frontoparallel plane. While this is a highly specific situation in a natural environment, it is also a situation that is especially difficult for human subjects and clearly benefits from extraretinal input as indicated in the psychophysical literature. Section 3will present simulations and analytical results of the model behavior in this situation. The comparison with neurophysiological findings will concern three aspects: the ability of neurons to differentiate real motion in the world from visual motion induced by eye movements (Section 4.1), the visual responses of optic flow neurons when eye movements are superimposed (Section 4.2), and the influence of extraretinal input on optic flow selectivity (Section 4.3). The model yields testable predictions for neurophysiological experiments.
Section snippets
The model
The goal of this work is to provide a neural network model of a specific functionality of the primate visual system. For the scope of this paper, we strive for biological plausibility which comprises three requirements. First (I), all computations have to be performed by neuronal elements that have only a limited processing power, like e.g. simple input–output neurons. Second (II), the model needs to conform with known anatomical and physiological properties of those parts of the visual system
Extraretinal input disambiguates flow field information
During movement with respect to a plane, the instantaneous optic flow field is inherently ambiguous (Longuet-Higgins, 1984). A particular situation occurs when an observer approaches a frontoparallel plane while performing an eye movement that is intended to stabilize gaze on a single point on the frontoparallel plane (Fig. 3). When such an optic flow field is simulated, human subjects cannot accurately determine their direction of heading (Regan and Beverly, 1982). Instead, they report the
Consequences of extraretinal input for single neurons in the model of area MST
Single neurons in the second layer of the model receive extraretinal input from the pursuit neurons in addition to the visual input they receive from the first layer neurons. The consequences of the interaction between the two signals for the response properties of individual second layer neurons were tested in simulations of single neurons. When only the visual input is considered, neurons in the second layer of the model respond selectively to various optic flow patterns (Lappe and
Discussion
Research in human optic flow processing has shown that the visual system benefits from extraretinal eye movement signals. These signals aid in the process of the detection of the direction of heading during combined self-motion and eye movements. In this paper a neural network model of heading detection was developed that uses both optic flow and extraretinal eye movement signals. This model takes as input a retinal flow field that results from combined self-motion and eye movements. It uses an
Acknowledgements
This research was funded by the Deutsche Forschungsgemeinschaft
References (74)
- et al.
Passive navigation.
Comput. Vis. Graph. Image Proc.
(1983) - et al.
Modification of the Filehne illusion by conditioning visual stimuli
Vision Res.
(1996) Recovering heading for visually-guided navigation
Vision Res.
(1992)- et al.
Coding of visual stimulus velocity in area MT of the macaque
Vision Res.
(1987) Analysis of misperceived observer motion during simulated eye rotations
Vision Res.
(1994)- et al.
Estimating heading during eye movements
Vision Res.
(1994) Robustness of perception of heading from optic flow
Vision Res.
(1992)Judgements of heading
Vision Res.
(1996)Centrifugal directionality bias in the middle temporal visual area (MT) of the macaque
Vis. Neurosci.
(1989)- et al.
Translational invariance and attentional modulation of MST cells
Soc. Neurosci. Abstr.
(1990)