Elsevier

Neural Networks

Volume 11, Issue 3, April 1998, Pages 397-414
Neural Networks

Contributed article
A model of the combination of optic flow and extraretinal eye movement signals in primate extrastriate visual cortex: Neural model of self-motion from optic flow and extraretinal cues

https://doi.org/10.1016/S0893-6080(98)00013-6Get rights and content

Abstract

The determination of the direction of heading from optic flow is a complicated task. To solve it the visual system complements the optic flow by non-visual information about the occurrence of eye movements. Psychophysical studies have shown that the need for this combination depends on the structure the visual scene. In a depth-rich visual environment motion parallax can be exploited to differentiate self-translation from eye rotation. In the absence of motion parallax, i.e. in the case of movement towards a frontoparallel plane, extraretinal signals are necessary for correct heading perception (Warren and Hannon, 1990). Lappe and Rauschecker (1993b) have proposed a model of visual heading detection that reproduces many of the psychophysical findings in the absence of extraretinal input and links them to properties of single neurons in the primate visual cortex. The present work proposes a neural network model that integrates extraretinal signals into this network. The model is compared with psychophysical and neurophysiological data from experiments in human and non-human primates. The combined visual/extraretinal model reproduces human behavior in the case of movement towards a frontoparallel plane. Single model neurons exhibit several similarities to neurons from the medial superior temporal (MST) area of the macaque monkey. Similar to MST cells (Erickson and Thier, 1991) they differentiate between self-induced visual motion that results from eye movements in a stationary environment, and real motion in the environment. The model predicts that this differentiation can also be achieved visually, i.e. without extraretinal input. Other simulations followed experiments by Bradley et al. (1996), in which flow fields were presented that simulated observer translation towards a frontoparallel plane plus an eye rotation. Similar to MST cells, model neurons shift their preference for the focus of expansion along the direction of the eye movement when extraretinal input is not available. They respond to the retinal location of the focus of expansion which is shifted by the eye movement. In the presence of extraretinal input the preference for the focus of expansion is largely invariant to eye movements and tied to the location of the focus of expansion with regard to the visual scene. The model proposes that extraretinal compensation for eye movements need not be perfect in single neurons to achieve accurate heading detection. It thereby shows that the incomplete compensation found in most MST neurons is sufficient to explain the psychophysical data.

Introduction

In higher mammals and humans, the perception of space and of motion through space is largely based on vision. Movements of the eyes in the head massively change the visual input though we continue to perceive a stable world around us. Questions about the use of extraretinal signals for the perception of such spatial stability during eye movements have long been raised and much discussed. Two classical theories present the two opposing approaches to this issue. The `inferential' theory proposes that, in order to attain stability of the visual world during eye movements, an extraretinal signal, e.g. an efference copy of the motor command driving the eye muscles, is used to compensate for the visual effects of the eye movement (von Helmholtz, 1896; Mittelstaedt, 1990; Sperry, 1950: Von Holst and Mittelstaedt, 1950). An opposite point of view is taken by the theory of `direct perception' (Gibson, 1950). In this view, no extraretinal signal is required, and the perception of the movement of objects in the world or of the observer himself is solely mediated from the visual information alone. In recent years, however, it has become evident that both theoretical views were for one reason or another inappropriate, and that aspects from both sides have to be considered (see Wertheim (1994) for a review). One important point in this regard is that the extraretinal signal typically underestimates the speed of the eye movement (Wertheim, 1987; Honda, 1990) and operates with a variable gain (Haarmeier and Thier, 1996).

A special case of this discussion concerns the detection of the direction of heading in the presence of eye movements. Eye movements here refer to slow eye movements such as smooth pursuit or the slow phases of the optokinetic and vestibulo-ocular reflexes. Since the original suggestion by Gibson (1950) that the optic flow field, i.e. the visual motion induced by self-motion in a stable world, provides all necessary information for navigation in an unknown environment, several computational (Bruss and Horn, 1983; Heeger and Jepson, 1992; Hildreth, 1992; Koenderink and van Doorn, 1987; Longuet-Higgins and Prazdny, 1980; Rieger and Lawton, 1985) and psychophysical (Rieger and Toet, 1985; van den Berg, 1992; Warren et al., 1988; Warren and Hannon, 1990) studies have supported this view. In the psychophysical experiments, the performance of human subjects in two conditions were compared. In the first condition, an optic flow stimulus simulating observer translation was presented while the subjects tracked a small pursuit target with his eyes. In the second condition, the subject kept his eyes stationary, while the stimulus simulated the visual effect of the eye movement together with the observer translation. Often there were few differences between the two conditions. There are, however, experimental situations which do require extraretinal signals. Most notably, a movement towards a frontoparallel plane (Regan and Beverly, 1982; Warren and Hannon, 1990) gives rise to a characteristic misjudgment of the direction of heading when only optic flow is available as a cue. However, also for movements on top of a ground plane, extraretinal signals are required, depending on the type of the eye movement. When eye movements simulate the pursuit of a horizontally moving target, humans cannot correctly judge the direction of heading (Royden et al., 1994). When the same eye movements are actually performed, i.e. when extraretinal information is available, the judgments are correct. In contrast, when eye movements simulate the stabilization of gaze on an environmental point, no extraretinal signal is required for a correct perception (van den Berg, 1993). The different simulated eye movements in these two situations give rise to fundamentally different flow patterns on the retina (Lappe and Rauschecker, 1995b). The different results can be linked to this difference in structure of the optic flow and to the similarity of the stimulus to the optic flow experienced during movement on a curved path (Lappe and Rauschecker, 1994; Royden, 1994).

Taken together, the present results from psychophysical studies suggest that the human visual system can make use of the information available in the optic flow field to determine the direction of heading. However, extraretinal input can improve the performance in many cases and is especially required to disambiguate the visual information in certain problematic situations.

The medial superior temporal area, or area MST, is regarded as a cortical site where the determination of egomotion parameters might take place (Duffy and Wurtz, 1991a, Duffy and Wurtz, 1995; Lagae et al., 1994; Pekel et al., 1996). Lappe and Rauschecker proposed a neural network model of heading detection from optic flow that achieves the visual response properties of MST neurons from the output of MT-like neurons (Lappe and Rauschecker, 1993b) and links these properties to the psychophysics of human heading detection from optic flow (Lappe and Rauschecker, 1995a, Lappe and Rauschecker, 1995b). Recent experimental studies (Duffy and Wurtz, 1995; Pekel et al., 1996; Lappe et al., 1996) have confirmed the predictions from this model.

However, besides its specificity for optic flow analysis, area MST is also linked to eye movements. `Pursuit' or `visual tracking' neurons in MST, discharge during ongoing smooth pursuit (Erickson and Dow, 1989; Komatsu and Wurtz, 1988) and receive extraretinal signals about the occurrence of a pursuit eye movement (Newsome et al., 1988; Ilg and Thier, 1996b). While area MST is involved in the generation of smooth pursuit (Dürsteler and Wurtz, 1988; Komatsu and Wurtz, 1988), these neurons might also provide area MST with a signal about the presence of an eye movement (Erickson and Thier, 1991). As with the extraretinal input to the pursuit neurons (Newsome et al., 1988) this distinction is not found earlier in the visual motion pathway, neither in area MT (Erickson and Thier, 1991), nor in area V1 (Ilg and Thier, 1996a).

This article presents a model of heading detection that includes not only the processing of visual, but also of extraretinal signals. In order to account for the psychophysically observed capabilities of the human visual system in the task of heading detection, neither of the two classical theories outlined above, the `inferential' or the `direct' theory, is sufficient. Instead, the psychophysical results suggest a situation-dependent combination of visual and extraretinal signals. Royden (1994) has proposed that the visual system uses an extraretinal signal to remove the eye-movement-induced motion from the retinal flow field and subsequently estimate translation and rotation on a curved path. The model presented here follows a similar approach, since it also first removes the eye-movement induced visual motion and then analyses the resulting `purified' flow field. However, different from Royden (1994) the subsequent analysis of the purified flow field still allows for residual influences of eye movements that might result from imperfect compensation. Moreover, the model provides a neuronal basis for this proposed combination of visual and extraretinal signals. Also, the paper will not be concerned with motion on a curved path, but rather with self-translations along a straight line.

The proposed model assumes that an extraretinal signal is first used to estimate the eye movement induced retinal image motion and then compensate for it. Consistent with experimental evidence (Haarmeier and Thier, 1996; Honda, 1990; Wertheim, 1987) this compensation mechanism must be assumed to be incomplete since it accounts only for a certain proportion of eye speed. Therefore the resulting purified flow field is nevertheless processed by a heading detection mechanism that allows for the presence of eye rotation and takes visual evidence of eye movements into account. The visual heading detection is based on an earlier developed model (Lappe and Rauschecker, 1993b). This earlier model does not include any extraretinal input but retrieves heading directions visually from retinal flow even in the presence of eye movements. It is successful in many situations and consistent with human psychophysical data in a way that the earlier model fails in the case of approaching a frontoparallel plane, in which extraretinal input is required. This paper describes how extraretinal input can be combined with this model such as to also retrieve heading in this more complicated situation. The interaction of visual and extraretinal signals in the model is based on known properties of MST neurons. The combination of visual and extraretinal signals is performed by optic flow processing MST-like model neurons that receive input from direction-selective MT-like model neurons and from MST-like model pursuit neurons. The basic mechanisms of this interaction, i.e. the requirements for the appropriate synaptic connections, have been briefly presented earlier (Lappe et al., 1994). This article will, after presenting the combined model in Section 2, compare it to psychophysical and neurophysiological data (3 Extraretinal input disambiguates flow field information, 4 Consequences of extraretinal input for single neurons in the model of area MST), and discuss the results and implications (Section 5). The comparison to psychophysical findings will focus on the case of an observer approaching a frontoparallel plane. While this is a highly specific situation in a natural environment, it is also a situation that is especially difficult for human subjects and clearly benefits from extraretinal input as indicated in the psychophysical literature. Section 3will present simulations and analytical results of the model behavior in this situation. The comparison with neurophysiological findings will concern three aspects: the ability of neurons to differentiate real motion in the world from visual motion induced by eye movements (Section 4.1), the visual responses of optic flow neurons when eye movements are superimposed (Section 4.2), and the influence of extraretinal input on optic flow selectivity (Section 4.3). The model yields testable predictions for neurophysiological experiments.

Section snippets

The model

The goal of this work is to provide a neural network model of a specific functionality of the primate visual system. For the scope of this paper, we strive for biological plausibility which comprises three requirements. First (I), all computations have to be performed by neuronal elements that have only a limited processing power, like e.g. simple input–output neurons. Second (II), the model needs to conform with known anatomical and physiological properties of those parts of the visual system

Extraretinal input disambiguates flow field information

During movement with respect to a plane, the instantaneous optic flow field is inherently ambiguous (Longuet-Higgins, 1984). A particular situation occurs when an observer approaches a frontoparallel plane while performing an eye movement that is intended to stabilize gaze on a single point on the frontoparallel plane (Fig. 3). When such an optic flow field is simulated, human subjects cannot accurately determine their direction of heading (Regan and Beverly, 1982). Instead, they report the

Consequences of extraretinal input for single neurons in the model of area MST

Single neurons in the second layer of the model receive extraretinal input from the pursuit neurons in addition to the visual input they receive from the first layer neurons. The consequences of the interaction between the two signals for the response properties of individual second layer neurons were tested in simulations of single neurons. When only the visual input is considered, neurons in the second layer of the model respond selectively to various optic flow patterns (Lappe and

Discussion

Research in human optic flow processing has shown that the visual system benefits from extraretinal eye movement signals. These signals aid in the process of the detection of the direction of heading during combined self-motion and eye movements. In this paper a neural network model of heading detection was developed that uses both optic flow and extraretinal eye movement signals. This model takes as input a retinal flow field that results from combined self-motion and eye movements. It uses an

Acknowledgements

This research was funded by the Deutsche Forschungsgemeinschaft

References (74)

  • R.W. Baloh et al.

    Eye movements induced by linear acceleration on a parallel swing

    J. Neurophysiol.

    (1988)
  • D. Boussaoud et al.

    Pathways for motion analysis: cortical connections of the medial superior temporal visual areas in the macaque

    J. Comp. Neurol.

    (1990)
  • D. Bradley et al.

    Mechanisms of heading perception in primate visual cortex

    Science

    (1996)
  • F. Bremmer et al.

    Eye position effects in monkey cortex. I: visual and pursuit related activity in extrastriate areas MT and MST

    J. Neurophysiol.

    (1997)
  • H. Bülthoff et al.

    A parallel algorithm for real-time computation of optical flow

    Nature

    (1989)
  • Carpenter, R.H.S., 1988. Movement of The Eyes, 2nd ed. Pion ,...
  • Duffy C.J., & Wurtz R.H. (1991a). Sensitivity of MST neurons to optic flow stimuli. I. A continuum of response...
  • Duffy C.J., & Wurtz R.H. (1991b). Sensitivity of MST neurons to optic flow stimuli. II. Mechanisms of response...
  • C.J. Duffy et al.

    Response of monkey MST neurons to optic flow stimuli with shifted centers of motion

    J. Neurosci.

    (1995)
  • M.R. Dürsteler et al.

    Pursuit and optokinetic deficits following chemical lesions of cortical areas MT and MST

    J. Neurophysiol.

    (1988)
  • R.G. Erickson et al.

    Foveal tracking cells in the superior temporal sulcus of the macaque monkey

    Exp. Brain Res.

    (1989)
  • R.G. Erickson et al.

    A neuronal correlate of spatial stability during periods of self-induced visual motion

    Exp. Brain Res.

    (1991)
  • R.G.L. Erickson et al.

    Responses of direction-selective neurons in monkey cortex to self-induced visual motion

    Ann. N.Y. Acad. Sci.

    (1992)
  • Gibson, J. J. (1950). The Perception of the Visual World. Houghton Mifflin,...
  • M.S.A. Graziano et al.

    Tuning of MST neurons to spiral motions

    J. Neurosci.

    (1994)
  • D.J. Heeger et al.

    Subspace methods for recovering rigid motion I: algorithm and implementation

    Int. J. Computer Vision

    (1992)
  • H. Honda

    The extraretinal signal from the pursuit eye movement system: Its role in the perceptual and the egocentric localization systems

    Percep. Psychophys.

    (1990)
  • Ilg U.J., & Thier P. (1996a). Inability of rhesus monkey area V1 to discriminate between self-induced and externally...
  • Ilg, U.J. & Thier, P. (1996b). MST neurons are activated by pursuit of imaginary targets. In: Thier, P., Karnath, H.-O....
  • K. Kawano et al.

    Response properties of neurons in posterior parietal cortex of monkey during visual-vestibular stimulation II. Optokinetic neurons

    J. Neurophysiol.

    (1984)
  • J.J. Koenderink et al.

    Invariant properties of the motion parallax field due to the movement of rigid bodies relative to an observer

    Opt. Acta

    (1975)
  • J.J. Koenderink et al.

    Facts on optic flow

    Biol. Cybernet.

    (1987)
  • H. Komatsu et al.

    Relation of cortical areas MT and MST to pursuit eye movements. I. Localization and visual properties of neurons

    J. Neurophysiol.

    (1988)
  • L. Lagae et al.

    Responses of macaque STS neurons to optic flow components: a comparison of areas MT and MST

    J. Neurophysiol.

    (1994)
  • Lappe, M., & Rauschecker, J. P. (1993a). Computation of heading direction from optic flow in visual cortex. In: Giles,...
  • Lappe, M., & Rauschecker, J. P. (1993b). A neural network for the processing of optic flow from egomotion in man and...
  • M. Lappe et al.

    Heading detection from optic flow

    Nature

    (1994)
  • Cited by (0)

    View full text