Abstract
Learning when and how to generalize knowledge from past experience to novel circumstances is a challenging problem many agents face. In animals, this generalization can be caused by mediated conditioning—when two stimuli gain a relationship through the mediation of a third stimulus. For example, in sensory preconditioning, if a light is always followed by a tone, and that tone is later paired with a shock, the light will come to elicit a fear reaction, even though the light was never directly paired with shock. In this paper, we present a computational model of mediated conditioning based on reinforcement learning with predictive representations. In the model, animals learn to predict future observations through the temporal-difference algorithm. These predictions are generated using both current observations and other predictions. The model was successfully applied to a range of animal learning phenomena, including sensory preconditioning, acquired equivalence, and mediated aversion. We suggest that animals and humans are fruitfully understood as representing their world as a set of chained predictions and propose that generalization in artificial agents may benefit from a similar approach.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Brogden, W.J.: Sensory pre-conditioning. Journal of Experimental Psychology 25, 323–332 (1939)
Rescorla, R.A.: Simultaneous and successive associations in sensory preconditioning. Journal of Experimental Psychology: Animal Behavior Processes 6, 207–216 (1980)
Honey, R.C., Hall, G.: Acquired equivalence and distinctiveness of cues. Journal of Experimental Psychology: Animal Behavior Processes 15, 338–346 (1989)
Hall, G.: Learning about associatively activated stimulus representations: Implications for acquired equivalence and perceptual learning. Animal Learning & Behavior 24, 233–255 (1996)
Holland, P.C.: Acquisition of representation-mediated conditioned food aversions. Learning & Motivation 12, 1–18 (1981)
Holland, P.C.: Event representation in pavlovian conditioning: Image and action. Cognition 37, 105–131 (1990)
Hall, G., Mitchell, C., Graham, S., Lavis, Y.: Acquired equivalence and distinctiveness in human discrimination learning: Evidence for associative mediation. Journal of Experimental Psychology: General 132, 266–276 (2003)
Littman, M.L., Sutton, R.S., Singh, S.: Predictive representations of state. In: Advances in Neural Information Processing Systems, vol. 14, pp. 1555–1561 (2002)
Rafols, E.J.R., Ring, M.B., Sutton, R.S., Tanner, B.: Using predictive representations to improve generalization in reinforcement learning. In: Proceedings of the International Joint Conference on Artificial Intelligence, pp. 835–840 (2005)
Sutton, R.S.: TD models: Modeling the world at a mixture of time scales. In: Proceedings of the 12th International Conference on Machine Learning, pp. 531–539 (1995)
Wagner, A.R.: SOP: A model of automatic memory processing in animal behavior. In: Spear, N.R., Miller, R.R. (eds.) Information processing in animals: Memory mechanisms, pp. 5–47. Erlbaum, Hillsdale (1981)
Gluck, M., Myers, C.: Hippocampal mediation of stimulus representation: A computational theory. Hippocampus 3, 491–516 (1993)
Sutton, R.S.: Learning to predict by the methods of temporal differences. Machine Learning 3, 9–44 (1988)
Bonardi, C., Rey, V., Richmond, M., Hall, G.: Acquired equivalence of cues in pigeon autoshaping: Effects of training with common consequences and with common antecedents. Animal Leaning & Behavior 21, 369–376 (1993)
Honey, R.C., Ward-Robinson, J.: Acquired equivalence and distinctiveness of cues: I. Exploring a neural network approach. Journal of Experimental Psychology: Animal Behavior Processes 28, 378–387 (2002)
Lawrence, D.H.: Acquired distinctiveness of cues: I. Transfer between discriminations on the basis of familiarity with the stimulus. Journal of Experimental Psychology 39, 770–784 (1949)
Myers, C.E., Shohamy, D., Gluck, M.A., Grossman, S., Kluger, A., Ferris, S., Golomb, J., Schnirman, G., Schwartz, R.: Dissociating hippocampal versus basal ganglia contributions to learning and transfer. Journal of Cognitive Neuroscience 15, 185–193 (2003)
Coutureau, E., Killcross, A.S., Good, M., Marshall, V.J., Ward-Robinson, J., Honey, R.C.: Acquired equivalence and distinctiveness of cues: II. Neural manipulations and their implications. Journal of Experimental Psychology: Animal Behavior Processes 28, 388–396 (2002)
Lazzaro, S.C., Gournani, K., Ludvig, E.A., Gluck, M.A.: Lesions of the entorhinal cortex abolish sensory preconditioning in rats. In: Society for Neuroscience Abstracts, 997.12 (2005)
Nicholson, D., Freeman, J.: Lesions of the perirhinal cortex impair sensory preconditioning in rats. Behavioral Brain Research 112, 69–75 (2000)
Port, R.L., Patterson, M.M.: Fimbrial lesions and sensory preconditioning. Behavioral Neuroscience 98, 584–589 (1984)
Schultz, W., Dayan, P., Montague, P.R.: A neural substrate of prediction and reward. Science 275, 1593–1599 (1997)
Koop, A.: Understanding experience: Temporal coherence and empirical knowledge representation. Master’s thesis, University of Alberta (2007)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ludvig, E.A., Koop, A. (2008). Learning to Generalize through Predictive Representations: A Computational Model of Mediated Conditioning. In: Asada, M., Hallam, J.C.T., Meyer, JA., Tani, J. (eds) From Animals to Animats 10. SAB 2008. Lecture Notes in Computer Science(), vol 5040. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-69134-1_34
Download citation
DOI: https://doi.org/10.1007/978-3-540-69134-1_34
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-69133-4
Online ISBN: 978-3-540-69134-1
eBook Packages: Computer ScienceComputer Science (R0)