Abstract
One of the fundamental aspects of human intelligence is the ability to process temporal information (Lashley, 1951). Learning and reproducing temporal sequences are closely associated with our ability to perceive and generate body movements, speech and language, music, etc. A considerable body of neural network literature is devoted to temporal pattern generation (see Wang, 2001, for a recent review). These models generally treat a temporal pattern as a sequence of discrete patterns, called a temporal sequence. Most of the models are based on either multilayer perceptrons with backpropagation training or the Hopfield model of associative recall. The basic idea for the former class of models is to view a temporal sequence as a set of associations between consecutive components, and learn these associations as input-output transformations (Jordan, 1986; Elman, 1990; Mozer, 1993). To deal with temporal dependencies beyond consecutive components, part of the input layer is used to keep a trace of history, behaving as short-term memory (STM). Similarly, for temporal recall based on the Hopfield associative memory, a temporal sequence is viewed as associations between consecutive components. These associations are stored in extended versions of the Hopfield model that includes some time delays (Sompolinsky & Kanter, 1986; Buhmann & Schulten, 1987; Heskes & Gielen, 1992). To deal with longer temporal dependencies, high-order networks have been proposed (Guyon et al., 1988).
Thanks to X. Liu for his help in typesetting. The preparation of this chapter was supported in part by an ONR YIP award and a grant from NUWC.
Preview
Unable to display preview. Download preview PDF.
References
Baram, Y. (1994). Memorizing binary vector sequences by a sparsely encoded network. IEEE Transactions on Neural Networks, 5(6), 974–981.
Barnes, J. M., & Underwood, B. J. (1959). ‘Fate’ of first-list associations in transfer theory. Journal of Experimental Psychology, 58, 97–105.
Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166.
Bower, G. H., Thompson-Schill, S., & Tulving, E. (1994). Reducing retroactive interference: An interference analysis. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 51–66.
Bradski, G., Carpenter, G. A., & Grossberg, S. (1994). STORE working memory networks for storage and recall of arbitrary temporal sequences. Biological Cybernetics, 71, 469–480.
Buhmann, J., & Schulten, K. (1987). Noise-driven temporal association in neural networks. Europhysics Letters, 4, 1205–1209.
Carpenter, G. A., & Grossberg, S. (1987). A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphs, and Imaging Processing, 37, 54–115.
Chandler, C. C. (1993). Accessing related events increases retroactive interference in a matching test. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 967–974.
Cohen, A., Ivry, R. I., & Keele, S. W. (1990). Attention and structure in sequence learning. Journal of Experimenal Psychology, 16, 17–30.
Cohen, M. A., & Grossberg, S. (1987). Masking fields: A massively parallel neural architecture for learning, recognizing, and predicting multiple groupings of patterned data. Applied Optics, 26, 1866–1891.
Crooks, R. L., & Stein, J. (1991). Psychology: Science, behavior, and life. Fort Worth, TX: Holt, Rinehart and Winston.
Curran, T., & Keele, S. W. (1993). Attentional and nonattentional forms of sequence learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 189–202.
Diederich, S., & Opper, M. (1987). Learning of correlated patterns in spin-like glass networks by local learning rules. Physical Review Letters, 58, 949–952.
Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14, 179–211.
French, R. M. (1994). Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference. In Proceedings of the Sixteenth Annual Conference of the Cognitive Science Society, (pp. 335–340). Hillsdale, NJ: Erlbaum.
Granger, R., Whitson, J., Larson, J., & Lynch, G. (1994). Non-Hebbian properties of long-term potentiation enable high-capacity encoding of temporal sequences. Proceedings of the National Academy of Sciences of USA, 91, 10104–10108.
Grossberg, S. (1976). Adaptive pattern classification and universal recoding: I. Parallel development and coding of neural feature detectors. Biological Cybernetics, 23, 121–134.
Grossberg, S. (1987). Competitive learning: From interactive activation to adaptive resonance. Cognitive Science, 11, 23–63.
Guyon, I., Personnaz, L., Nadal, J. P., & Dreyfus, G. (1988). Storage and retrieval of complex sequences in neural networks. Physics Review A, 38, 6365–6372.
Hebb, D. O. (1949). The Organization of behavior. New York: Wiley & Sons.
Hertz, H., Krogh, A., & Palmer, R. G. (1991). Introduction to the theory of neural computation. Redwood City, CA: Addison-Wesley.
Heskes, T. M., & Gielen, S. (1992). Retrieval of pattern sequences at variable speeds in a neural network with delays. Neural Networks, bf 5, 145–152.
Jordan, M. I. (1986). Attractor dynamics and parallelism in a connectionist sequential machine. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, (pp. 531–546). Hillsdale, NJ: Erlbaum.
Kanerva, P. (1988). Sparse distributed memory. Cambridge, MA: MIT Press.
Kantor, I., & Sompolinsky, H. (1987). Associative recall of memory without errors. Physics Review A, bf 35, 380–392.
Kruschke, J. K. (1992). ALCOVE: An exemplar-based model of category learning. Psychological Review, 99, 22–44.
Lashley, K. S. (1951). The problem of serial order in behavior. In L. A. Jeffress (Ed.), Cerebral mechanisms in behavior (pp. 112–146). New York: Wiley & Sons.
Lewandowsky, S., & Murdock Jr., B. B. (1989). Memory for serial order. Psychological Review, 96, 25–57.
McCloskey, M., & Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24, 109–165.
Meyer, L. B. (1956). Emotion and meaning in music. Chicago, IL: University of Chicago Press.
Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81–97.
Mozer, M. C. (1993). Neural net architectures for temporal sequence processing. In A. Weigend & N. Gershenfeld (Ed.), Predicting the future and understanding the past (pp. 243–264). Redwood City, CA: Addison-Wesley.
Murdock, B. B. J. (1987). Serial-order effects in a distributed-memory model. In D. S. Gorfein & R. R. Hoffman (Ed.), Memory and learning: The Ebbinghaus centennial conference (pp. 227–310). Hillsdale, NJ: Erlbaum.
Nissen, M. J., & Bullemer, P. (1987). Attentional requirements of learning: Evidence from performance measures. Cognitive Psychology, 19, 1–32.
Ratcliff, R. (1990). Connectionist models of recognition memory: Constraints imposed by learning and forgetting function. Psychological Review, 97, 285–308.
Rinkus, G. J. (1995). TEMECOR: an associative, spatio-temporal pattern memory for complex state sequences. In Proceedings of World Congress on Neural Networks, (pp. I.442–I.448). Washington DC
Rodriguez, W. A., Borbely, L. S., & Garcia, R. S. (1993). Attenuation by contextual cues of retroactive interference of a conditional discrimination in rats. Animal Learning & Behavior, 21, 101–105.
Ruiz de Angulo, V., & Torras, C. (1995). On-line learning with minimal degradation in feedforward networks. IEEE Transactions on Neural Networks, 6, 657–668.
Simon, H. A. (1974). How big is a chunk? Science, 183, 482–488.
Sloman, S. A., & Rumelhart, D. E. (1992). Reducing interference in distributed memories through episodic gating. In A. F. Healy, S. M. Kosslyn, & R. M. Shiffrin (Ed.), From learning theory to connectionist theory: Essays in honor of William K. Estes (pp. 227–248). Hillsdale, NJ: Erlbaum.
Sompolinsky, H., & Kanter, I. (1986). Temporal association in asymmetric neural networks. Physics Review Letters, 57, 2861–2864.
von der Malsburg, C. (1973). Self-organization of orientation sensitive cells in the striate cortex. Kybernetik, 14, 85–100.
Wang, D. L. (2001). Temporal pattern processing. In M. A. Arbib (Ed.), Handbook of brain theory and neural networks, Second Edition (to appear). Cambridge MA: MIT Press.
Wang, D. L., & Arbib, M. A. (1990). Complex temporal sequence learning based on short-term memory. Proceedings of the IEEE, 78, 1536–1543.
Wang, D. L., & Arbib, M. A. (1993). Timing and chunking in processing temporal order. IEEE Transactions on Systems, Man, and Cybernetics, 23, 993–1009.
Wang, D. L., & Yuwono, B. (1995). Anticipation-based temporal pattern generation. IEEE Transactions on Systems, Man, and Cybernetics, 25, 615–628.
Wang, D. L., & Yuwono, B. (1996). Incremental learning of complex temporal patterns. IEEE Transactions on Neural Networks, 7, 1465–1481.
Wang, L. (1999). Multi-associative neural networks and their applications to learning and retrieving complex spatio-temporal sequences. IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics, 29, 73–82.
Willingham, D. B., Nissen, M. J., & Bullemer, P. (1989). On the development of procedural knowledge. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15, 1047–1060.
Willshaw, D. J., Buneman, O. P., & Longuet-Higgins, H. C. (1969). Nonholographic associative memory. Nature, 222, 960–962.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2000 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Wang, D. (2000). Anticipation Model for Sequential Learning of Complex Sequences. In: Sun, R., Giles, C.L. (eds) Sequence Learning. Lecture Notes in Computer Science(), vol 1828. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44565-X_4
Download citation
DOI: https://doi.org/10.1007/3-540-44565-X_4
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-41597-8
Online ISBN: 978-3-540-44565-4
eBook Packages: Springer Book Archive