Skip to main content

Anticipation Model for Sequential Learning of Complex Sequences

  • Chapter
  • First Online:
Sequence Learning

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 1828))

  • 1081 Accesses

Abstract

One of the fundamental aspects of human intelligence is the ability to process temporal information (Lashley, 1951). Learning and reproducing temporal sequences are closely associated with our ability to perceive and generate body movements, speech and language, music, etc. A considerable body of neural network literature is devoted to temporal pattern generation (see Wang, 2001, for a recent review). These models generally treat a temporal pattern as a sequence of discrete patterns, called a temporal sequence. Most of the models are based on either multilayer perceptrons with backpropagation training or the Hopfield model of associative recall. The basic idea for the former class of models is to view a temporal sequence as a set of associations between consecutive components, and learn these associations as input-output transformations (Jordan, 1986; Elman, 1990; Mozer, 1993). To deal with temporal dependencies beyond consecutive components, part of the input layer is used to keep a trace of history, behaving as short-term memory (STM). Similarly, for temporal recall based on the Hopfield associative memory, a temporal sequence is viewed as associations between consecutive components. These associations are stored in extended versions of the Hopfield model that includes some time delays (Sompolinsky & Kanter, 1986; Buhmann & Schulten, 1987; Heskes & Gielen, 1992). To deal with longer temporal dependencies, high-order networks have been proposed (Guyon et al., 1988).

Thanks to X. Liu for his help in typesetting. The preparation of this chapter was supported in part by an ONR YIP award and a grant from NUWC.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Baram, Y. (1994). Memorizing binary vector sequences by a sparsely encoded network. IEEE Transactions on Neural Networks, 5(6), 974–981.

    Article  Google Scholar 

  • Barnes, J. M., & Underwood, B. J. (1959). ‘Fate’ of first-list associations in transfer theory. Journal of Experimental Psychology, 58, 97–105.

    Article  Google Scholar 

  • Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166.

    Article  Google Scholar 

  • Bower, G. H., Thompson-Schill, S., & Tulving, E. (1994). Reducing retroactive interference: An interference analysis. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 51–66.

    Article  Google Scholar 

  • Bradski, G., Carpenter, G. A., & Grossberg, S. (1994). STORE working memory networks for storage and recall of arbitrary temporal sequences. Biological Cybernetics, 71, 469–480.

    Article  Google Scholar 

  • Buhmann, J., & Schulten, K. (1987). Noise-driven temporal association in neural networks. Europhysics Letters, 4, 1205–1209.

    Article  Google Scholar 

  • Carpenter, G. A., & Grossberg, S. (1987). A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphs, and Imaging Processing, 37, 54–115.

    Article  Google Scholar 

  • Chandler, C. C. (1993). Accessing related events increases retroactive interference in a matching test. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 967–974.

    Article  Google Scholar 

  • Cohen, A., Ivry, R. I., & Keele, S. W. (1990). Attention and structure in sequence learning. Journal of Experimenal Psychology, 16, 17–30.

    Google Scholar 

  • Cohen, M. A., & Grossberg, S. (1987). Masking fields: A massively parallel neural architecture for learning, recognizing, and predicting multiple groupings of patterned data. Applied Optics, 26, 1866–1891.

    Article  Google Scholar 

  • Crooks, R. L., & Stein, J. (1991). Psychology: Science, behavior, and life. Fort Worth, TX: Holt, Rinehart and Winston.

    Google Scholar 

  • Curran, T., & Keele, S. W. (1993). Attentional and nonattentional forms of sequence learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 189–202.

    Article  Google Scholar 

  • Diederich, S., & Opper, M. (1987). Learning of correlated patterns in spin-like glass networks by local learning rules. Physical Review Letters, 58, 949–952.

    Article  MathSciNet  Google Scholar 

  • Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14, 179–211.

    Article  Google Scholar 

  • French, R. M. (1994). Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference. In Proceedings of the Sixteenth Annual Conference of the Cognitive Science Society, (pp. 335–340). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Granger, R., Whitson, J., Larson, J., & Lynch, G. (1994). Non-Hebbian properties of long-term potentiation enable high-capacity encoding of temporal sequences. Proceedings of the National Academy of Sciences of USA, 91, 10104–10108.

    Article  Google Scholar 

  • Grossberg, S. (1976). Adaptive pattern classification and universal recoding: I. Parallel development and coding of neural feature detectors. Biological Cybernetics, 23, 121–134.

    Article  MATH  MathSciNet  Google Scholar 

  • Grossberg, S. (1987). Competitive learning: From interactive activation to adaptive resonance. Cognitive Science, 11, 23–63.

    Article  Google Scholar 

  • Guyon, I., Personnaz, L., Nadal, J. P., & Dreyfus, G. (1988). Storage and retrieval of complex sequences in neural networks. Physics Review A, 38, 6365–6372.

    Article  MathSciNet  Google Scholar 

  • Hebb, D. O. (1949). The Organization of behavior. New York: Wiley & Sons.

    Google Scholar 

  • Hertz, H., Krogh, A., & Palmer, R. G. (1991). Introduction to the theory of neural computation. Redwood City, CA: Addison-Wesley.

    Google Scholar 

  • Heskes, T. M., & Gielen, S. (1992). Retrieval of pattern sequences at variable speeds in a neural network with delays. Neural Networks, bf 5, 145–152.

    Google Scholar 

  • Jordan, M. I. (1986). Attractor dynamics and parallelism in a connectionist sequential machine. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, (pp. 531–546). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Kanerva, P. (1988). Sparse distributed memory. Cambridge, MA: MIT Press.

    MATH  Google Scholar 

  • Kantor, I., & Sompolinsky, H. (1987). Associative recall of memory without errors. Physics Review A, bf 35, 380–392.

    Google Scholar 

  • Kruschke, J. K. (1992). ALCOVE: An exemplar-based model of category learning. Psychological Review, 99, 22–44.

    Article  Google Scholar 

  • Lashley, K. S. (1951). The problem of serial order in behavior. In L. A. Jeffress (Ed.), Cerebral mechanisms in behavior (pp. 112–146). New York: Wiley & Sons.

    Google Scholar 

  • Lewandowsky, S., & Murdock Jr., B. B. (1989). Memory for serial order. Psychological Review, 96, 25–57.

    Article  Google Scholar 

  • McCloskey, M., & Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24, 109–165.

    Article  Google Scholar 

  • Meyer, L. B. (1956). Emotion and meaning in music. Chicago, IL: University of Chicago Press.

    Google Scholar 

  • Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81–97.

    Article  Google Scholar 

  • Mozer, M. C. (1993). Neural net architectures for temporal sequence processing. In A. Weigend & N. Gershenfeld (Ed.), Predicting the future and understanding the past (pp. 243–264). Redwood City, CA: Addison-Wesley.

    Google Scholar 

  • Murdock, B. B. J. (1987). Serial-order effects in a distributed-memory model. In D. S. Gorfein & R. R. Hoffman (Ed.), Memory and learning: The Ebbinghaus centennial conference (pp. 227–310). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Nissen, M. J., & Bullemer, P. (1987). Attentional requirements of learning: Evidence from performance measures. Cognitive Psychology, 19, 1–32.

    Article  Google Scholar 

  • Ratcliff, R. (1990). Connectionist models of recognition memory: Constraints imposed by learning and forgetting function. Psychological Review, 97, 285–308.

    Article  Google Scholar 

  • Rinkus, G. J. (1995). TEMECOR: an associative, spatio-temporal pattern memory for complex state sequences. In Proceedings of World Congress on Neural Networks, (pp. I.442–I.448). Washington DC

    Google Scholar 

  • Rodriguez, W. A., Borbely, L. S., & Garcia, R. S. (1993). Attenuation by contextual cues of retroactive interference of a conditional discrimination in rats. Animal Learning & Behavior, 21, 101–105.

    Google Scholar 

  • Ruiz de Angulo, V., & Torras, C. (1995). On-line learning with minimal degradation in feedforward networks. IEEE Transactions on Neural Networks, 6, 657–668.

    Article  Google Scholar 

  • Simon, H. A. (1974). How big is a chunk? Science, 183, 482–488.

    Article  Google Scholar 

  • Sloman, S. A., & Rumelhart, D. E. (1992). Reducing interference in distributed memories through episodic gating. In A. F. Healy, S. M. Kosslyn, & R. M. Shiffrin (Ed.), From learning theory to connectionist theory: Essays in honor of William K. Estes (pp. 227–248). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Sompolinsky, H., & Kanter, I. (1986). Temporal association in asymmetric neural networks. Physics Review Letters, 57, 2861–2864.

    Article  Google Scholar 

  • von der Malsburg, C. (1973). Self-organization of orientation sensitive cells in the striate cortex. Kybernetik, 14, 85–100.

    Article  Google Scholar 

  • Wang, D. L. (2001). Temporal pattern processing. In M. A. Arbib (Ed.), Handbook of brain theory and neural networks, Second Edition (to appear). Cambridge MA: MIT Press.

    Google Scholar 

  • Wang, D. L., & Arbib, M. A. (1990). Complex temporal sequence learning based on short-term memory. Proceedings of the IEEE, 78, 1536–1543.

    Article  Google Scholar 

  • Wang, D. L., & Arbib, M. A. (1993). Timing and chunking in processing temporal order. IEEE Transactions on Systems, Man, and Cybernetics, 23, 993–1009.

    Article  Google Scholar 

  • Wang, D. L., & Yuwono, B. (1995). Anticipation-based temporal pattern generation. IEEE Transactions on Systems, Man, and Cybernetics, 25, 615–628.

    Article  Google Scholar 

  • Wang, D. L., & Yuwono, B. (1996). Incremental learning of complex temporal patterns. IEEE Transactions on Neural Networks, 7, 1465–1481.

    Article  Google Scholar 

  • Wang, L. (1999). Multi-associative neural networks and their applications to learning and retrieving complex spatio-temporal sequences. IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics, 29, 73–82.

    Article  Google Scholar 

  • Willingham, D. B., Nissen, M. J., & Bullemer, P. (1989). On the development of procedural knowledge. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15, 1047–1060.

    Article  Google Scholar 

  • Willshaw, D. J., Buneman, O. P., & Longuet-Higgins, H. C. (1969). Nonholographic associative memory. Nature, 222, 960–962.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2000 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Wang, D. (2000). Anticipation Model for Sequential Learning of Complex Sequences. In: Sun, R., Giles, C.L. (eds) Sequence Learning. Lecture Notes in Computer Science(), vol 1828. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44565-X_4

Download citation

  • DOI: https://doi.org/10.1007/3-540-44565-X_4

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-41597-8

  • Online ISBN: 978-3-540-44565-4

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics