Abstract
Two of the most promising aspects of connectionist natural language research have been (i) the use of powerful statistical learning techniques to model language learning and (ii) the development of new representational theories. Often the two are treated together; some part of a grammar is induced by a net and the subsequent representations are analysed for the maintenance of structural information. In this chapter, representation and learning are treated separately. A simple recurrent net trained on a bidirectional link grammar showed severe limitations in its ability to handle embedded sequences. Then, after an analysis of the problem, a constructive method was used to develop representations, using the same SRN architecture, that exhibited the potential to correctly recognise embeddings of any length. These findings illustrate the benefits of the study of representation, which can provide a basis for the development of novel learning rules.
We are grateful to ESRC Grant No R-000-22-1133 for funding this research, and to Stuart Jackson for running the simulations, and for his contribution to the development of the ideas on which this work is based.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
References
Blank, D.S., Meeden, L.A., & Marshall, J.B. (1992) Exploring the symbolic/subsymbolic continuum: A case study of RAAM. In J.Dinsmore, (Ed) The Symbolic and Connectionist Paradigms: Closing the Gap, Lawrence Erlbaum Associates.
Chalmers, D.J. (1990) Syntactic transformations on distributed representations. Connection Science, 2(1), 53–62.
Charniak, E. (1995) Learning syntactic disambiguation through word statistics and why you should care about it, Paper presented to the IJCAI-95 Workshop on New Approaches to Learning for Natural Language Processing,August
Chrisman, L. (1991) Learning recursive distributed representations for holistic computation. Connection Science 3.4, 345–366.
Cleeremans, A. (1993) Mechanisms of implicit learning: Connectionist models of Sequence Processing. MIT Press; London, England.
Denker, J., Schwartz, D., Wittner, B., Solla, S., Howard, R., Jackel, L., & Hopfield, J. (1987) Large automatic learning, rule extraction, and generalisation. Complex Systems, 1, 877–922.
Elman, J.L. (1988) Finding structure in time (Tech. Rep. No. 8801) San Diego, CA: University of California, Center for Research in Language.
Elman, J.L. (1990) Finding structure in time. Cognitive Science, 14, 179–211.
Elman, J.L. (1991) Distributed representations, simple recurrent networks and grammatical structure. Machine Learning, 7, 195–225.
Fodor, J.A. & McLaughlin, B.P. (1990) Connectionism and the problem of systematicity: why Smolensky's solution doesn't work. Cognition, 35, 183–204
Fodor, J.A. & Pylyshyn, Z. (1988) Connectionism and cognitive architecture: A critical analysis. Cognition, 28, 3–71.
Hadley, R. F., (1994), Systematicity in Connectionist Language Learning, In Mind and Language, vol. 9, no. 3, Blackwell Publishers.
Hinton, G.E., (1981) Implementing semantic networks in parallel hardware. In G.E. Hinton & J.A. Anderson (Eds) Parallel models of associative memory. Hillsdale, N.J.: Lawrence Erlbaum Associates Inc. 161–187.
Hinton, G.E. (1986) Learning distributed representations of concepts. Proceedings of Eighth Annual Conference of the Cognitive Science Society
McClelland, J.L. & Rumelhart, D.E. (1981) An interactive activation model of context effects in letter perception: Part 1. An account of basic findings. Psychological Review, 88, 375–407.
Niklasson, L. & Sharkey, N.E. (1992) Connectionism and the issues of compositionality and systematicity. In R. Trappl (Ed) Cybernetics and Systems. Dordrecht: Kluwer Academic Press.
Niklasson, L.F. & van Gelder, T. (1994) Can connectionist models exhibit non-classical structure sensitivity. Proceedings of the Cognitive Science Society, pp 664–669.
Plunkett, K., & Marchman, V.A. (1991) U-shaped learning and frequency effects in a multi-layered perceptron: Implications for child language acquisition. Cognition, 38, 43–102.
Pollack, J. (1990) Recursive distributed representations. Artificial Intelligence, 46, 77–105.
Rumelhart, D.E., Hinton, G.E., and Williams, R.J. (1986) Learning internal representations by error propogation, In D.E. Rumelhart and J.L. McClelland, Parallel Distributed Processing: Explorations in the microstructure of cognition, Vol 1: Foundations, MIT Press: Cambridge, MA, chapter 8.
Rumelhart, D.E., McClelland, J.L., & the PDP Research Group (1986) On learning the past tenses of English verbs. In D.E. Rumelhart & J.L. McClelland (Eds) Parallel Distributed Processing: Explorations in the microstructure of cognition. Vol 1. Cambridge, MA: MIT Press
Sejnowski, T.J., & Rosenberg, C. (1986) NetTalk: A parallel network that learns to read aloud. Tech. Rep. JHU-EECS-86-01, Johns Hopkins University.
Servan-Schreiber, D., Cleeremans, A., & McClelland, J.L. (1991) Graded state machines: The representation of temporal contingencies in simple recurrent networks. Machine Learning, 7, 161–193.
Sharkey, N.E. (1991) Connectionist representation techniques. Artificial Intelligence Review, 5, 143–167.
Sharkey, N.E. (Ed) (1991) Connectionist Natural Language Processing. Oxford:Intellect and Kluwer.
Sharkey, N.E. and Jackson, S.A. (1994) An Internal report for Connectionists In R. Sun and L. Bookman (Eds) Computational Architecture's Integrating Neural and Symbolic Processes. Kluwer: MA, 223–244
Sharkey, N.E. & Reilly, R.G. (1992) Connectionist Natural Language Processing In R.G. Reilly and N.E. Sharkey (Eds) Connectionist Approaches to Natural Language Processing. Lawrence Erlbaum Associates: Hillsdale, N.J. pp 1–9.
Sharkey, N.E. & Sharkey, A.J.C. (1992) A modular design for connectionist parsing. Connectionism and Natural Language Processing. M.F.J. Drosaers & A. Nijholt (Eds) Proceedings of Workshop on Language Technology 3, Twente, 87–96.
Sharkey, N.E. & Sharkey, A.J.C. (1993) Adaptive generalization. Artificial Intelligence Review 7, 313–328.
Sharkey, N.E., Sharkey, A.J.C., and Jackson, S.A. (1994) Opening the black box of Connectionist Nets: Some lessons from Cognitive Science, Computer Standards and Interfaces, 16, 279–293
Smolensky, P. (1988) On the proper treatment of connectionism. The Behavioural and Brain Sciences, 11.
van Gelder, T. (1990) Compositionality: A Connectionist variation on a classical theme. Cognitive Science, 14, pp 355–364.
White, H. (1992) Artificial Neural Networks: Approximation and Learning theory. Blackwell. Oxford.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1996 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Sharkey, N.E., Sharkey, A.J.C. (1996). Separating learning and representation. In: Wermter, S., Riloff, E., Scheler, G. (eds) Connectionist, Statistical and Symbolic Approaches to Learning for Natural Language Processing. IJCAI 1995. Lecture Notes in Computer Science, vol 1040. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-60925-3_35
Download citation
DOI: https://doi.org/10.1007/3-540-60925-3_35
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-60925-4
Online ISBN: 978-3-540-49738-7
eBook Packages: Springer Book Archive