Abstract
This article explores how the effectiveness of learning to parse with neural networks can be improved by including two architectural features relevant to language: generalisations across syntactic constituents and bounded resource effects. A number of neural network parsers have recently been proposed, each with a different approach to the representational problem of outputting parse trees. In addition, some of the parsers have explicitly attempted to capture an important regularity within language, which is to generalise information across syntactic constituents. A further property of language is that natural bounds exist for the number of constituents which a parser need retain for later processing. Both the generalisations and the resource bounds may be captured in architectural features which enhance the effectiveness and efficiency of learning to parse with neural networks. We describe a number of different types of neural network parser, and compare them with respect to these two features. These features are both explicitly present in the Simple Synchrony Network parser, and we explore and illustrate their impact on the process of learning to parse in some experiments with a recursive grammar.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
E. Charniak, Statistical Language Learning, MIT Press: Cambridge, MA, 1993.
M. Johnson, “PCFG models of linguistic tree representations,” Computational Linguistics, vol. 24, pp. 613–632, 1998.
R.F. Hadley and M.B. Hayward, “Strong semantic systematicity from Hebbian connectionist learning,” Minds and Machines, vol. 7, pp. 1–37, 1997.
J.B. Henderson, “A neural network parser that handles sparse data,” in Proceedings of the 6th International Workshop on Parsing Technologies, Trento, Italy, 2000, pp. 123–134.
J.B. Henderson and P.C.R. Lane, “A connectionist architecture for learning to parse,” in Proceedings of the 17th International Conference on Computational Linguistics and the 36 th Annual Meeting of the Association for Computational Linguistics (COLING-ACL’98), University of Montreal, Canada, 1998, pp. 531–537.
E.K.S. Ho and L.W. Chan, “How to design a connectionist holistic parser,” Neural Computation, vol. 11, no.8, pp. 1995–2016, 1999.
P.C.R. Lane, “Simple Synchrony Networks: A new connectionist architecture applied to natural language parsing,” Ph.D. Thesis, Department of Computer Science, University of Exeter, England, 2000.
P.C.R. Lane and J.B. Henderson, “Simple Synchrony Networks: Learning to parse natural language with Temporal Synchrony Variable Binding,” in Proceedings of the Eighth International Conference on Artificial Neural Networks, edited by L. Niklasson, M. Boden, and T. Ziemke, Skövde, Sweden, 1998, pp. 615–620.
P.C.R. Lane and J.B. Henderson, “Incremental syntactic parsing of natural language corpora with Simple Synchrony Networks,” IEEE Transactions on Knowledge and Data Engineering, vol. 13, pp. 219–231, 2001.
R. Miikkulainen, “Subsymbolic case-role analysis of sentences with embedded clauses,” Cognitive Science, vol. 20, pp. 47–73, 1996.
J.A. Fodor and Z.W. Pylyshyn, “Connectionism and cognitive architecture: A critical analysis,” Cognition, vol. 28, pp. 3–71, 1988.
N. Cowan, “The magical number 4 in short-term memory: A reconsideration of mental storage capacity,” Behavioral and Brain Sciences, vol. 24, no.1, pp. 87–185, 2001.
G.A. Miller, “The magical number seven, plus or minus two: Some limits on our capacity for processing information,” Psychological Review, vol. 63, pp. 81–97, 1956.
N. Chomsky, Aspects of the Theory of Syntax, MIT Press: Cambridge, MA, 1965.
J. Kimball, “Seven principles of surface structure parsing in natural language,” Cognition, vol. 2, pp. 15–47, 1976.
M. Marcus, A Theory of Syntactic Recognition for Natural Language, MIT Press: Cambridge, MA, 1980.
D. Chalmers, “Syntactic transformations of distributed representations,” in Connectionist Natural Language Processing, edited by N. Sharkey, Kluwer: Boston, 1992, pp. 46–55.
L. Chrisman, “Learning recursive distributed representations for holistic computation,” Connection Science, vol. 3, pp. 345–366, 1991.
R.F. Hadley, “Systematicity in connectionist language learning,” Mind and Language, vol. 9, pp. 247–272, 1994.
R.F. Hadley, “Systematicity revisited: Reply to Christiansen and Chater and Niklasson and van Gelder,” Mind and Language, vol. 9, pp. 431–444, 1994.
K. Aizawa, “Exhibiting versus explaining systematicity: A reply to Hadley and Hayward,” Minds and Machines, vol. 7, pp. 39–55, 1997.
L. Shastri and V. Ajjanagadde, “From simple associations to systematic reasoning: A connectionist representation of rules, variables, and dynamic bindings using temporal synchrony,” Behavioral and Brain Sciences, vol. 16, pp. 417–494, 1993.
J.B. Henderson, “A connectionist architecture with inherent systematicity,” in Proceedings of the Eighteenth Conference of the Cognitive Science Society, La Jolla, CA, 1996, pp. 574–579.
E. Gibson, “A computational theory of human linguistic processing: Memory limitations and processing breakdown,” Ph.D. Thesis, Carnegie Mellon University, Pittsburgh, PA, 1991.
K. Hornik, W. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, pp. 359–366, 1989.
E.K.S. Ho and L.W. Chan, “Confluent preorder parsing of deterministic grammars,” Connection Science, vol. 9, pp. 269–293, 1997.
R. Callan and D. Palmer-Brown, “An analytical technique for fast and reliable derivation of connectionist symbol structure representations,” Connection Science, vol. 9, no.2, pp. 139–159, 1997.
J. Pollack, “Recursive distributed representations,” Artificial Intelligence, vol. 46, pp. 77–105, 1990.
D.E. Rumelhart, G.E. Hinton, and R.J. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, edited by D.E. Rumelhart, J.L. McClelland, and the PDP Research Group, MIT Press: Cambridge, MA, vol. 1, 1986.
J.B. Henderson, “Description based parsing in a connectionist network,” Ph.D. Thesis, University of Pennsylvania, 1994.
J.L. Elman, “Distributed representations, simple recurrent networks, and grammatical structure,” Machine Learning, vol. 7, pp. 195–225, 1991.
R. Miikkulainen, “Natural language processing with subsymbolic neural networks,” in Neural Network Perspectives on Cognition and Adaptive Robotics, edited by A. Browne, 1997, pp. 120–139.
P.C.R. Lane, “Simple Synchrony Networks: Learning generalisations across syntactic constituents,” in Proceedings of the Thirteenth European Conference in Artificial Int elligence, Brighton: John Wiley & Sons, UK, 1998, pp. 469–470.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Lane, P.C., Henderson, J.B. Towards Effective Parsing with Neural Networks: Inherent Generalisations and Bounded Resource Effects. Applied Intelligence 19, 83–99 (2003). https://doi.org/10.1023/A:1023820807862
Issue Date:
DOI: https://doi.org/10.1023/A:1023820807862