Skip to main content
SpringerLink
Log in

Learning visuomotor transformations for gaze-control and grasping

  • Published:
Biological Cybernetics Aims and scope Submit manuscript

Abstract

For reaching to and grasping of an object, visual information about the object must be transformed into motor or postural commands for the arm and hand. In this paper, we present a robot model for visually guided reaching and grasping. The model mimics two alternative processing pathways for grasping, which are also likely to coexist in the human brain. The first pathway directly uses the retinal activation to encode the target position. In the second pathway, a saccade controller makes the eyes (cameras) focus on the target, and the gaze direction is used instead as positional input. For both pathways, an arm controller transforms information on the target’s position and orientation into an arm posture suitable for grasping. For the training of the saccade controller, we suggest a novel staged learning method which does not require a teacher that provides the necessary motor commands. The arm controller uses unsupervised learning: it is based on a density model of the sensor and the motor data. Using this density, a mapping is achieved by completing a partially given sensorimotor pattern. The controller can cope with the ambiguity in having a set of redundant arm postures for a given target. The combined model of saccade and arm controller was able to fixate and grasp an elongated object with arbitrary orientation and at arbitrary position on a table in 94% of trials.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • AP Batista CA Buneo LH Snyder RA Andersen (1999) ArticleTitleReach plans in eye-centered coordinates Science 285 257–260 Occurrence Handle10.1126/science.285.5425.257 Occurrence Handle10398603

    Article  PubMed  Google Scholar 

  • GG Blasdel G Salama (1986) ArticleTitleVoltage-sensitive dyes reveal a modular organization in monkey striate cortex Nature 321 579–585 Occurrence Handle10.1038/321579a0 Occurrence Handle3713842

    Article  PubMed  Google Scholar 

  • B Bridgeman (1995) ArticleTitleA review of the role of efference copy in sensory and oculomotor control systems Ann Biomed Eng 23 409–422 Occurrence Handle7486348

    PubMed  Google Scholar 

  • CA Buneo MR Jarvis AP Batista RA Andersen (2002) ArticleTitleDirect visuomotor transformations for reaching Nature 416 632–636 Occurrence Handle10.1038/416632a Occurrence Handle11948351

    Article  PubMed  Google Scholar 

  • DP Carey RJ Coleman SD Sala (1997) ArticleTitleMagnetic misreaching Cortex 33 639–652 Occurrence Handle9444466

    PubMed  Google Scholar 

  • DP Carey SD Sala M Ietswaart (2002) Neuropsychological perspectives on eye-hand coordination in visually-guided reaching. In: Progress in brain research vol 140 Elsevier Amsterdam 311–327

    Google Scholar 

  • R Cipolla N Hollinghurst (1997) ArticleTitleVisually guided grasping in unstructured environments Rob Auton Syst 19 337–346 Occurrence Handle10.1016/S0921-8890(96)00060-7

    Article  Google Scholar 

  • JD Crawford D Guitton (1997) ArticleTitleVisual-motor transformations required for accurate and kinematically correct saccades J Neurophysiol 78 1447–1467 Occurrence Handle9310435

    PubMed  Google Scholar 

  • JD Crawford WP Medendorp JJ Marotta (2004) ArticleTitleSpatial transformation for eye-hand coordination J Neurophysiol 92 10–19 Occurrence Handle10.1152/jn.00117.2004 Occurrence Handle15212434

    Article  PubMed  Google Scholar 

  • H Cruse U Steinkühler (1993) ArticleTitleSolution of the direct and inverse kinematic problems by a common algorithm based on the mean of multiple computations Biol Cybern 69 345–351 Occurrence Handle10.1007/BF00203131

    Article  Google Scholar 

  • KI Diamantaras SY Kung (1996) Principal component neural networks Wiley New York

    Google Scholar 

  • C Distante A Anglani F Taurisano (2000) ArticleTitleTarget reaching by using visual information and Q-learning controllers Auton Robots 9 41–50 Occurrence Handle10.1023/A:1008972101435

    Article  Google Scholar 

  • O Fuentes RC Nelson (1998) ArticleTitleLearning dextrous manipulation skills for multifingered robot hands using the evolution strategy Mach Learn 31 223–237 Occurrence Handle10.1023/A:1007409228154

    Article  Google Scholar 

  • MS Graziano CS Taylor T Moore (2002) ArticleTitleComplex movements evoked by microstimulation of precentral cortex Neuron 34 841–851 Occurrence Handle10.1016/S0896-6273(02)00698-0 Occurrence Handle12062029

    Article  PubMed  Google Scholar 

  • GE Hinton P Dayan M Revow (1997) ArticleTitleModeling the manifolds of images of handwritten digits IEEE Trans Neural Netw 8 IssueID1 65–74 Occurrence Handle10.1109/72.554192

    Article  Google Scholar 

  • H Hoffmann R Möller (2003) Unsupervised learning of a kinematic arm model O Kaynak E Alpaydin E Oja L Xu (Eds) Artificial neural networks and neural information processing – ICANN/ICONIP, LNCS, vol 2714 Springer Berlin Heidelberg New York 463–470

    Google Scholar 

  • JJ Hopfield (1982) ArticleTitleNeural networks and physical systems with emergent collective computational abilities Proc Nat Acad Sci USA 79 2554–2558 Occurrence Handle6953413

    PubMed  Google Scholar 

  • DH Hubel TN Wiesel (1962) ArticleTitleReceptive fields, binocular interaction and functional architecture in the cat’s visual cortex J Physiol 160 106–154 Occurrence Handle14449617

    PubMed  Google Scholar 

  • MI Jordan DE Rumelhart (1992) ArticleTitleForward models: supervised learning with a distal teacher Cogn Sci 16 307–354 Occurrence Handle10.1016/0364-0213(92)90036-T

    Article  Google Scholar 

  • M Kawato (1990) Feedback-error-learning neural network for supervised motor learning R Eckmiller (Eds) Advanced neural computers Elsevier North-Holland 365–372

    Google Scholar 

  • M Kawato K Furukawa R Suzuki (1987) ArticleTitleA hierarchical neural-network model for control and learning of voluntary movement Biol Cybern 57 169–185 Occurrence Handle10.1007/BF00364149 Occurrence Handle3676355

    Article  PubMed  Google Scholar 

  • EM Klier JD Crawford (1998) ArticleTitleHuman oculomotor system accounts for 3-D eye orientation in the visual-motor transformation for saccades J Neurophysiol 80 2274–2294 Occurrence Handle9819243

    PubMed  Google Scholar 

  • M Kuperstein (1988) ArticleTitleNeural model of adaptive hand-eye coordination for single postures Science 239 1308–1311 Occurrence Handle3344437

    PubMed  Google Scholar 

  • M Kuperstein (1990) ArticleTitleINFANT neural controller for adaptive sensory-motor coordination Neural Netw 4 131–145 Occurrence Handle10.1016/0893-6080(91)90001-L

    Article  Google Scholar 

  • F Lacquaniti R Caminiti (1998) ArticleTitleVisuo-motor transformations for arm reaching Eur J Neurosci 10 195–203 Occurrence Handle10.1046/j.1460-9568.1998.00040.x Occurrence Handle9753127

    Article  PubMed  Google Scholar 

  • PE Latham S Deneve A Pouget (2003) ArticleTitleOptimal computation with attractor networks J Physiol 97 683–694

    Google Scholar 

  • RJ Leigh DS Zee (1999) The neurology of eye movements Oxford University Press New York

    Google Scholar 

  • TM Martinetz SG Berkovich KJ Schulten (1993) ArticleTitle‘‘Neural-Gas’’ network for vector quantization and its application to time-series prediction IEEE Trans Neural Netw 4 IssueID4 558–569 Occurrence Handle10.1109/72.238311

    Article  Google Scholar 

  • JP Miller GA Jacobs FE Theunissen (1991) ArticleTitleRepresentation of sensory information in the cricket cercal sensory system I. Response properties of the primary interneurons J Neurophysiol 66 IssueID5 1680–1689 Occurrence Handle1765801

    PubMed  Google Scholar 

  • J Molina-Vilaplana JL Pedreño-Molina J López-Coronado (2004) ArticleTitleHyper RBF model for accurate reaching in redundant robotic systems Neurocomputing 61 495–501 Occurrence Handle10.1016/j.neucom.2004.06.006

    Article  Google Scholar 

  • R Möller H Hoffmann (2004) ArticleTitleAn extension of neural gas to local PCA Neurocomputing 62 305–326 Occurrence Handle10.1016/j.neucom.2003.09.014

    Article  Google Scholar 

  • JR Movellan JL McClelland (1993) ArticleTitleLearning continuous probability distributions with symmetric diffusion networks Cogn Sci 17 463–496 Occurrence Handle10.1016/0364-0213(93)90001-O

    Article  Google Scholar 

  • S Ouyang Z Bao GS Liao (2000) ArticleTitleRobust recursive least squares learning algorithm for principal component analysis IEEE Trans Neural Netw 11 IssueID1 215–221 Occurrence Handle10.1109/72.822524

    Article  Google Scholar 

  • E Oztop NS Bradley MA Arbib (2004) ArticleTitleInfant grasp learning: a computational model Exp Brain Res 158 480–503 Occurrence Handle10.1007/s00221-004-1914-1 Occurrence Handle15221160

    Article  PubMed  Google Scholar 

  • Riedmiller M, BraunH(1993)A direct adaptive method for faster backpropagation learning: the RPROP algorithm. In: Proceedings of the IEEE international conference on neural networks. San Francisco, pp 586–591

  • HJ Ritter (1993) Parametrized self-organizing maps. In: Gielen S Kappen B (eds) Proceedings of the international conference on artificial neural networks Springer Berlin Heidelberg New York 568–575

    Google Scholar 

  • HJ Ritter TM Martinetz KJ Schulten (1989) ArticleTitleTopology-conserving maps for learning visuo-motor-coordination Neural Netw 2 159–168 Occurrence Handle10.1016/0893-6080(89)90001-4

    Article  Google Scholar 

  • G Rizzolatti L Fadiga (1998) ArticleTitleGrasping objects and grasping action meanings: the dual role of monkey rostroventral premotor cortex (area F5) Novartis Found Symp 218 81–103 Occurrence Handle9949817

    PubMed  Google Scholar 

  • GL Ruskell (1999) ArticleTitleExtraocular muscle proprioceptors and proprioception Prog Retin Eye Res 18 269–291 Occurrence Handle10.1016/S1350-9462(98)00029-9 Occurrence Handle10192514

    Article  PubMed  Google Scholar 

  • M Salganicoff LH Ungar R Bajcsy (1996) ArticleTitleActive learning for vision-based robot grasping Mach Learn 23 251–278

    Google Scholar 

  • W Schenck R Möller (2004) Staged learning of saccadic eye movements with a robot camera head H Bowman C Labiouse (Eds) Connectionist models of cognition and perception II World Scientific London 82–91

    Google Scholar 

  • Schenck W, Hoffmann H, Möller R (2003) Learning internal models for eye-hand coordination in reaching and grasping. In: Proceedings of the European cognitive science conference, Erlbaum, Mahwah, pp 289-294

  • LH Snyder (2000) ArticleTitleCoordinate transformations for eye and arm movements in the brain Curr Opin Neurobiol 10 747–754 Occurrence Handle10.1016/S0959-4388(00)00152-5 Occurrence Handle11240284

    Article  PubMed  Google Scholar 

  • LH Snyder AP Batista RA Andersen (2000) ArticleTitleSaccade-related activity in the parietal reach region J Neurophysiol 83 1099–1102 Occurrence Handle10669521

    PubMed  Google Scholar 

  • DL Sparks LE Mays (1990) ArticleTitleSignal transformations required for the generation of saccadic eye movements Ann Rev Neurosci 13 309–336 Occurrence Handle10.1146/annurev.ne.13.030190.001521 Occurrence Handle2183679

    Article  PubMed  Google Scholar 

  • U Steinkühler H Cruse (1998) ArticleTitleA holistic model for an internal representation to control the movement of a manipulator with redundant degrees of freedom Biol Cybern 79 457–466 Occurrence Handle10.1007/s004220050495

    Article  Google Scholar 

  • ME Tipping CM Bishop (1999) ArticleTitleMixtures of probabilistic principal component analyzers Neural Comput 11 443–482 Occurrence Handle10.1162/089976699300016728 Occurrence Handle9950739

    Article  PubMed  Google Scholar 

  • S Treue JCM Trujillo (1999) ArticleTitleFeature-based attention influences motion processing gain in macaque visual cortex Nature 399 575–579 Occurrence Handle10.1038/21176 Occurrence Handle10376597

    Article  PubMed  Google Scholar 

  • Umbaugh SE (1998) Computer vision and image processing: a practical approach using CVIPtools. Prentice Hall.

  • Y Uno N Fukumura R Suzuki M Kawato (1995) ArticleTitleA computational model for recognizing objects and planning hand shapes in grasping movements Neural Netw 8 839–851 Occurrence Handle10.1016/0893-6080(95)00002-H

    Article  Google Scholar 

  • Walter JA, Nölker C, Ritter H (2000) The PSOM algorithm and applications. In: Proceedings of the symposium on neural computation, pp 758–764

  • DM Wolpert Z Ghahramani MI Jordan (1995) ArticleTitleAre arm trajectories planned in kinematic or dynamic coordinates? An adaptation study Exp Brain Res 103 460–470 Occurrence Handle10.1007/BF00241505 Occurrence Handle7789452

    Article  PubMed  Google Scholar 

  • D Zipser RA Andersen (1988) ArticleTitleA back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons Nature 331 679–684 Occurrence Handle10.1038/331679a0 Occurrence Handle3344044

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Max Planck Institute for Human Cognitive and Brain Sciences, Cognitive Robotics, 80799, Munich, Germany

    Heiko Hoffmann

  2. Computer Engineering Group, Faculty of Technology, Bielefeld University, 33594, Bielefeld, Germany

    Wolfram Schenck & Ralf Möller

Authors
  1. Heiko Hoffmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wolfram Schenck
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ralf Möller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Heiko Hoffmann.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hoffmann, H., Schenck, W. & Möller, R. Learning visuomotor transformations for gaze-control and grasping. Biol Cybern 93, 119–130 (2005). https://doi.org/10.1007/s00422-005-0575-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00422-005-0575-x

Keywords

Search

Navigation