Skip to main content
SpringerLink
Log in

A Hybrid Architecture for Situated Learning of Reactive Sequential Decision Making

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In developing autonomous agents, one usually emphasizes only (situated) procedural knowledge, ignoring more explicit declarative knowledge. On the other hand, in developing symbolic reasoning models, one usually emphasizes only declarative knowledge, ignoring procedural knowledge. In contrast, we have developed a learning model CLARION, which is a hybrid connectionist model consisting of both localist and distributed representations, based on the two-level approach proposed in [40]. CLARION learns and utilizes both procedural and declarative knowledge, tapping into the synergy of the two types of processes, and enables an agent to learn in situated contexts and generalize resulting knowledge to different scenarios. It unifies connectionist, reinforcement, and symbolic learning in a synergistic way, to perform on-line, bottom-up learning. This summary paper presents one version of the architecture and some results of the experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Sutton and Barto (1981).

  2. A. Schultz, “Using a genetic algorithm to learn strategies for collisionavoidance and local navigation,” in Proc. of 7th International Symp. on Unmanned Untethered Submersible Technology, University of New Hampshire, Durham, pp. 213-225, 1991.

    Google Scholar 

  3. R. Sun and T. Peterson, “A hybrid learning model of reactive sequential decision making,” in The Working Notes of The IJCAI Workshop on Connectionist-Symbolic Integration, edited by R. Sun and F. Alexandre, 1995.

  4. P. Rosenbloom, J. Laird, and A. Newell, The SOAR Papers: Research on Integrated Intelligence, MIT Press: Cambridge, MA, 1993.

    Google Scholar 

  5. R. Nosofsky, T. Palmeri, and S. McKinley, “Rule-plus-exception model of classification learning,” Psychological Review, vol. 101,no. 1, pp. 53-79, 1994.

    Google Scholar 

  6. G. Widmer and M. Kubat, “Learning in the presence of concept drift and hidden context,” Machine Learning, vol. 23,no. 1, 1996.

  7. R. Sutton, “Learning to predict by the methods of temporal difference,” Machine Learning, vol. 3, pp. 9-44, 1988.

    Google Scholar 

  8. R. Sutton, “Integrated architectures for learning, planning, and reacting based on approximating dynamic programming,” in Proc. of Seventh International Conference on Machine Learning, Morgan Kaufmann: San Mateo, CA, 1990.

    Google Scholar 

  9. S. Mahadevan and J. Connell, “Automatic programming of behavior-based robot with reinforcement learning,” vol. 55, pp. 311-365, 1992.

    Google Scholar 

  10. L. Lin, “Self-improving reactive agents based on reinforcement learning, planning, and teaching,” Machine Learning, vol. 8, pp. 293-321, 1992.

    Google Scholar 

  11. L. Kaelbling, M. Littman, and A. Moore, “Reinforcement learning: A survey,” Journal of Artificial Intelligence Research, vol. 4, pp. 237-285, 1996.

    Google Scholar 

  12. T. Dietterich, Hierarchical reinforcement learning with MAXQ value function decomposition, 1997, ftp://www.cs.orst.edu.

  13. J. Holland, N. Nisbitt, P. Thagard, and J. Holyoak, Induction: A Theory of Learning and Development, MIT Press: Cambridge, MA, 1986.

    Google Scholar 

  14. J. Grefenstette, “The evolution of strategies for multiagent environments,” Adaptive Behavior, vol. 1,no. 1, pp. 65-90, 1992.

    Google Scholar 

  15. L. Meeden, “An incremental approach to developing intelligent neural network controllers for robots,” Adaptive Behavior, 1995.

  16. R. Michalski, “A theory and methodology of inductive learning,” Artificial Intelligence, vol. 20, pp. 111-161, 1983.

    Google Scholar 

  17. R. Quinlan, “Inductive learning of decision trees,” Machine Learning, vol. 1, pp. 81-106, 1986.

    Google Scholar 

  18. R. Quinlan, “Learning logical definition from relations,” Machine Learning, vol. 5, pp. 239-266, 1990.

    Google Scholar 

  19. T. Mitchell, “Generalization as search,” Artificial Intelligence, vol. 18, pp. 203-226, 1982.

    Google Scholar 

  20. M. Lebowitz, “Experiments with incremental concept formation: UNIMEM,” Machine Learning, vol. 2, pp. 103-138, 1987.

    Google Scholar 

  21. D. Fisher, “Knowledge acquisition via incremental conceptual clustering,” Machine Learning, vol. 2, pp. 139-172, 1987.

    Google Scholar 

  22. P. Utgoff, “Incremental induction of decision trees,” Machine Learning, vol. 4, pp. 161-186, 1989.

    Google Scholar 

  23. H. Hirsh, “Generalizing version spaces,” Machine Learning, vol. 17, pp. 5-46, 1994.

    Google Scholar 

  24. P. Clark and T. Niblett, “The CN2 induction algorithm,” Machine Learning, vol. 3, pp. 261-284, 1989.

    Google Scholar 

  25. R. Sun and T. Peterson, “Some experiments with a hybrid model for learning sequential decision making,” Information Sciences, vol. 14, pp. 83-107, 1998.

    Google Scholar 

  26. R. Sun and T. Peterson, “Autonomous learning of sequential tasks: Experiments and analyses,” IEEE Transaction on Neural Networks, vol. 9,no. 6, pp. 1217-1234, 1998.

    Google Scholar 

  27. J. Anderson, “Acquisition of cognitive skill,” Psychological Review, vol. 89, pp. 369-406, 1982.

    Google Scholar 

  28. J. Anderson, Rules of the Mind, Lawrence Erlbaum Associates: Hillsdale, NJ, 1993.

    Google Scholar 

  29. J. Anderson and C. Lebiere, The Atomic Components of Thought, Lawrence Erlbaum Associates: Mahwah, NJ, 1998.

    Google Scholar 

  30. F. Keil, Concepts, Kinds, and Cognitive Development, MIT Press: Cambridge, MA, 1989.

    Google Scholar 

  31. A. Damasio et al., “Neural regionalization of knowledge access,” Cold Spring Harbor Symp. on Quantitative Biology, CSHL Press, vol. LV, 1990.

  32. R. Sun, Integrating Rules and Connectionism for Robust Commonsense Reasoning, John Wiley and Sons: New York, NY, 1994.

    Google Scholar 

  33. J. Anderson, The Architecture of Cognition, Harvard University Press: Cambridge, MA, 1983.

    Google Scholar 

  34. P. Fitts and M. Posner, Human Performance, Brooks/Cole, Monterey, CA, 1967.

    Google Scholar 

  35. H. Dreyfus and S. Dreyfus, Mind Over Machine, The Free Press: New York, NY, 1987.

    Google Scholar 

  36. P. Smolensky, “On the proper treatment of connectionism,” Behavioral and Brain Sciences, vol. 11,no. 1, pp. 1-74, 1988.

    Google Scholar 

  37. W. James, The Principles of Psychology, Dover: New York, 1890.

    Google Scholar 

  38. R. Sun, “Learning, action, and consciousness: A hybrid approach towards modeling consciousness,” Neural Networks, special issue on consciousness, vol. 10,no. 7, pp. 1317-1331, 1997.

    Google Scholar 

  39. D. Willingham, M. Nissen, and P. Bullemer, “On the development of procedural knowledge,” Journal of Experimental Psychology: Learning, Memory, and Cognition, vol. 15, pp. 1047-1060, 1989.

    Google Scholar 

  40. R. Sun, “Robust reasoning: Integrating rule-based and similarity-based reasoning,” Artificial Intelligence, vol. 75,no. 2, pp. 241-296, 1995.

    Google Scholar 

  41. J. Hendler, “Marker passing and microfeature,” in Proc. 10th IJCAI, Morgan Kaufmann: San Mateo, CA, 1987, pp. 151-154.

    Google Scholar 

  42. J. Gelfand, D. Handelman, and S. Lane, “Integrating knowledge-based systems and neural networks for robotic skill acquisition,” in Proc. IJCAI, Morgan kaufmann: San Mateo, CA, 1989, pp. 193-198.

    Google Scholar 

  43. W. Schneider and W. Oliver, “An instructable connectionist/control architecture,” in Architectures for Intelligence, edited by K. VanLehn, Erlbaum: Hillsdale, NJ, 1991.

    Google Scholar 

  44. R. Sun, “A Connectionist model for commonsense reasoning incorporating rules and similarities,” Knowledge Acquisition, vol. 4, pp. 293-321, 1992.

    Google Scholar 

  45. M. Erickson and J. Kruschke, Rules and Examplars in Category Learning, manuscript, 1997.

  46. A. Reber, “Implicit learning and tacit knowledge,” Journal of Experimental Psychology: General, vol. 118,no. 3, pp. 219-235, 1989.

    Google Scholar 

  47. R. Dominowski, How do People Discover Concepts?, manuscript, 1975.

  48. D. Medin, W. Wattenmaker, and R. Michalski, “Constraints and preferences in inductive learning: An experimental study of human and machine performance,” Cognitive Science, vol. 11, pp. 299-339, 1987.

    Google Scholar 

  49. C. Watkins, “Learning with Delayed Rewards,” Ph.D. Thesis, Cambridge University, Cambridge, UK, 1989.

    Google Scholar 

  50. D. Bertsekas and J. Tsitsiklis, Neuro-Dynamic Programming, Athena Scientific: Belmont, MA, 1996.

    Google Scholar 

  51. R. Parr and S. Russell, “Reinforcement learning with hierarchies of machines,” Advances in Neural Information Processing Systems, MIT Press: Cambridge, MA, 1997.

    Google Scholar 

  52. D. Precup, R. Sutton, and S. Singh, “Multi-time models for temporary abstract planning,” Advances in Neural Information Processing Systems 10, MIT Press: Cambridge, MA, 1998.

    Google Scholar 

  53. T. Tesauro, “Practical issues in temporal difference learning,” Machine Learning, vol. 8, pp. 257-277, 1992.

    Google Scholar 

  54. J. Boyan and A. Moore, “Generalization in reinforcement learning: Safely approximating the value function,” in Neural Information Processing Systems, edited by J. Tesauro and D. Touretzky, and T. Leen, MIT Press: Cambridge, MA, pp. 369-376, 1995.

    Google Scholar 

  55. R. Sun, “On variable binding in connectionist networks,” Connection Science, vol. 4,no. 2, pp. 93-124, 1992.

    Google Scholar 

  56. L.M. Fu, “Rule learning by searching on adapted nets,” in Proc. of AAAI'91, 1991, pp. 590-595.

  57. R.C. Lacher, “Expert networks: Paradigmatic conflict, technological rapprochment,” Minds and Machines, vol 3, pp. 53-71, 1993.

    Google Scholar 

  58. G. Towell and J. Shavlik, “Extracting refined rules from knowledge-based neural networks,” Machine Learning, vol. 13,no. 1, pp. 71-101, 1993.

    Google Scholar 

  59. R. Michalski, I. Mozetic, J. Hong, and N. Lavrac, “The multipurpose incremental learning system AQ15,” in Proc. of AAAI-86, Morgan Kaufmann: San Mateo, CA, 1986, pp. 1041-1045.

    Google Scholar 

  60. A. McCallum, “Learning to use selective attention and short-term memory in sequential tasks,” in Proc. Conference on Simulation of Adaptive Behavior, MIT Press: Cambridge, MA, 1996, pp. 315-324.

    Google Scholar 

  61. L. Breiman, “Bagging predictors,” Machine Learning, vol. 24,no. 2, pp. 123-140, 1996.

    Google Scholar 

  62. Y. Freund and R. Schapire, “Experiments with a new boosting algorithm,” in Proc. of ICML'97, Morgan Kaufmann, San Francisco, CA, 1996, pp. 148-156.

    Google Scholar 

  63. R. Brooks, “Intelligence without representation,” Artificial Intelligence, vol. 47, pp. 139-160, 1991.

    Google Scholar 

  64. P. Tadepalli and T. Dietterich, “Hierarchical explanation-based reinforcement learning,” in Proc. International Conference on Machine Learning, Morgan Kaufmann: San Francisco, CA, 1997, pp. 358-366.

    Google Scholar 

  65. D. Gordon, A. Schultz, J. Grefenstette, J. Ballas, and M. Perez, User's Guide to the Navigation and Collision Avoidance Task, Naval Research Lab, Washington, DC, 1994.

    Google Scholar 

  66. R. Sun, E. Merrill, and T. Peterson, “A bottom-up model of skill learning,” in Proc. of 20th Cognitive Science Society Conference, Lawrence Erlbaum Associates: Mahwah, NJ, pp. 1037-1042, 1998.

    Google Scholar 

  67. R. Sun, E. Merrill, and T. Peterson, “Skill learning using a bottom-up hybrid model,” in Proc. of The Second European Conference on Cognitive Modeling, Nottingham University Press: Nottingham, UK, pp. 23-29, April 1998.

    Google Scholar 

  68. E. Smith and D. Medin, Categories and Concepts, Cambridge, MA: Harvard University Press, 1981.

    Google Scholar 

  69. E. Rosch, “Principles of categorization,” in Cognition and Categorization, edited by E. Rosch and B. Lloyd, Erlbaum: Hillsdale, NJ, 1978.

    Google Scholar 

  70. C. Giraud-Carrier and T. Martinez, “An integrated framework for learning and reasoning,” Journal of Artificial Intelligence Research, vol. 3, pp. 147-185, 1995.

    Google Scholar 

  71. D. Waltz, “How to build a robot,” in Proc. of Conf. on Simulation of Adaptive Behaviors, edited by S. Wilson, MIT Press: Cambridge, MA, 1991.

    Google Scholar 

  72. R. Sun, E. Merrill, and T. Peterson, “A bottom-up model of skill learning,” in Proc. of 20th Cognitive Science Society Conference, Lawrence Erlbaum Associates: Mahwah, NJ, pp. 1037-1042, 1998.

    Google Scholar 

  73. M. Lebowitz, “Experiments with incremental concept formation: UNIMEM,” Machine Learning, vol. 2, pp. 103-138, 1987.

    Google Scholar 

  74. D. Fisher, “Knowledge acquisition via incremental conceptual clustering,” Machine Learning, vol. 2, pp. 139-172, 1987.

    Google Scholar 

  75. R. Stepp and R. Michalski, “Conceptual clustering,” in Machine Learning, II, edited by R. Michalski et al., Morgan Kaufmann: Los Altos, CA Stepp and Michalski (1983).

    Google Scholar 

  76. W. Shen, “Discovery as autonomous learning from the environment,” Machine Learning, vol. 12, pp. 143-165, 1993.

    Google Scholar 

  77. E. Wisniewski and D. Medin, “On the interaction of data and theory in concept learning,” Cognitive Science, vol. 18, pp. 221-281, 1994.

    Google Scholar 

  78. L. Rips, “Similarity, typicality, and categorization,” in Similarity and Analogical Reasoning, edited by S. Vosniadou and A. Ortony, Cambridge University Press: New York, NY, 1989.

    Google Scholar 

  79. R. Andrews and J. Diederich (Eds.), Proceedings of the NIPS'96 Workshop on Rule Extraction From Trained Artificial Neural Networks, NIPS Foundation, 1996.

  80. R. Miikkulainen and M. Dyer, “Natural language processing with modular PDP networks and distributed lexicons,” Cognitive Science, vol. 15,no. 3, pp. 343-399, 1991.

    Google Scholar 

  81. D. Gordon and D. Subramanian, “A cognitive model of learning to navigate,” in Proc. of 18th Cognitive Science Conference, Lawrence Erlbaum: Mahwah, NJ, 1997, pp. 271-276.

    Google Scholar 

  82. T. Johnson, J. Zhang, and H. Wang, “A hybrid learning model of abductive reasoning,” in Connectionist-Symbolic Integration, edited by R. Sun and F. Alexandre, Lawrence Erlbaum Associates: Mahwah, NJ, 1998.

    Google Scholar 

  83. V. Ajjanagadde and L. Shastri, “From simple association to systematic reasoning,” Tech. Report MS-CIS-90-05, University of Pennsylvania, Philadelphia, PA, 1990.

    Google Scholar 

  84. J. Barnden, “The right of free association: Relative-position encoding for connectionist data structures,” in Proc. 10th Conference of Cognitive Science Society, Lawrence Erlbaum Associates: Hillsdale, NJ, pp. 503-509, 1988.

    Google Scholar 

  85. J. Kruschke, “ALCOVE: an examples-based connectionist model of category learning,” Psychological Review, vol. 99, pp. 22-44, 1992.

    Google Scholar 

  86. D. Touretzky and G. Hinton, “Symbols among neurons,” in Proc. 9th IJCAI, Morgan Kaufmann, 1987, pp. 238-243.

  87. C.L. Giles and M. Gori, “Adaptive processing of sequences and data structures,” Lecture Notes in Artificial Intelligence, Springer Verlag, 1998.

  88. J. Barnden, “The right of free association: Relative-position encoding for connectionist data structures,” in Proc. 10th Conference of Cognitive Science Society, Lawrence Erlbaum Associates: Hillsdale, NJ, pp. 503-509, 1988.

    Google Scholar 

  89. E. Gat, “Integrating planning and reacting in a heterogeneous architecture,” in Proc. AAAI, Morgan Kaufmann: San Mateo, CA, 1992, pp. 809-815.

    Google Scholar 

  90. W. Schneider and W. Oliver, “An instructable connectionist/control architecture,” in Architectures for Intelligence, edited by K. VanLehn, Erlbaum: Hillsdale, NJ, 1991.

    Google Scholar 

  91. R. Maclin and J. Shavlik, “Incorporating advice into agents that learn from reinforcements,” in Proc. of AAAI-94, Morgan Kaufmann: San Meteo, CA, 1994.

    Google Scholar 

  92. G. Drescher, Made-up Minds, MIT Press: Cambridge, MA, 1991.

    Google Scholar 

  93. D. Broadbent, P. Fitsgerald, and M. Broadbent, “Implicit and explicit knowledge in the control of complex systems,” British Journal of Psychology, vol. 77, pp. 33-50, 1986.

    Google Scholar 

  94. G. Logan, “Toward an instance theory of automatization,” Psychological Review, vol. 95,no. 4, pp. 492-527, 1988.

    Google Scholar 

  95. W. Stanley, R. Mathews, R. Buss, and S. Kotler-Cope, “Insight without awareness: On the interaction of verbalization, instruction and practice in a simulated process control task,” Quarterly Journal of Experimental Psychology, vol. 41A,no. 3, pp. 553-577, 1989.

    Google Scholar 

  96. D. Gordon and D. Subramanian, “A multistrategy learning scheme for agent knowledge acquisition,” Informatica, vol. 17, pp. 331-346, 1993.

    Google Scholar 

  97. E. Thorndike, Animal Intelligence, Hafner: Darien, Connecticutt, 1911.

    Google Scholar 

  98. K. VanLehn, “Cognitive skill acquisition,” in Annual Review of Psychology, edited by J. Spence, J. Darly, and D. Foss, Annual Reviews, Palo Alto, CA, vol. 47, pp. 513-539, 1996.

    Google Scholar 

  99. S. Whitehead and D. Ballard, “Learning to perceive and act by trial and error,” Artificial Intelligence, vol. 7, pp. 45-83, 1991.

    Google Scholar 

  100. J. Schlimmer, “Incremental adjustment of representations for learning,” in Proc.of 4th Workshop on Machine Learning, Morgan Kaufmann: San Mateo, CA, pp. 502-507, 1987.

    Google Scholar 

  101. P. Agre and D. Chapman, “What are plans for?,” in Designing Autonomous Agents, edited by P. Maes, Elsevier: New York, 1990.

    Google Scholar 

  102. A. Barto, R. Sutton, and C. Watkins, “Learning and sequential decision-making,” in Learning and Computational Neuroscience, edited by M. Gabriel and J. Moors, MIT Press: Cambridge, MA.

  103. J. LeDoux, “Brain mechanisms of emotion and emotional learning,” Current Opinion in Neurobiology, vol. 2,no. 2, pp. 191-197, 1992

    Google Scholar 

  104. R. Shiffrin and W. Schneider, “Controlled and automatic human information processing II,” Psychological Review, vol. 84, pp. 127-190, 1977.

    Google Scholar 

  105. R. Stepp and R. Michalski, “Conceptual clustering,” in Machine Learning, II, edited by R. Michalski et al., Morgan Kaufmann: Los Altos, CA, 1986.

    Google Scholar 

  106. R. Sun and L. Bookman (Eds.), Computational Architectures Integrating Neural and Symbolic Processes, Kluwer Academic Publishers: Norwell, MA, 1994.

    Google Scholar 

  107. D. Touretzky and G. Hinton, “Symbols among neurons,” in Proc. 9th IJCAI, Morgan Kaufmann, 1987, pp. 238-243.

  108. K. VanLehn, “Rule acquisiton events in the discovery of problem-solving strategies,” Cognitive Science, vol. 15, pp. 1-47, 1991.

    Google Scholar 

  109. S. Whitehead and L. Lin, “Reinforcement learning of non-Markov decision processes,” Artificial Intelligence, vol. 73,nos. 1–2, pp. 271-306, 1995.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. NEC Research Institute and University of Alabama, 4 Independence Way, Princeton, NJ, 08540

    Ron Sun

  2. The University of Alabama, Tuscaloosa, AL, 35487

    Todd Peterson & Edward Merrill

Authors
  1. Ron Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Todd Peterson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Edward Merrill
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sun, R., Peterson, T. & Merrill, E. A Hybrid Architecture for Situated Learning of Reactive Sequential Decision Making. Applied Intelligence 11, 109–127 (1999). https://doi.org/10.1023/A:1008332731824

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1008332731824

Search

Navigation