Skip to main content
SpringerLink
Log in

Multi-agent systems with reinforcement hierarchical neuro-fuzzy models

  • Published:
Autonomous Agents and Multi-Agent Systems Aims and scope Submit manuscript

Abstract

This paper introduces a new multi-agent model for intelligent agents, called reinforcement learning hierarchical neuro-fuzzy multi-agent system. This class of model uses a hierarchical partitioning of the input space with a reinforcement learning algorithm to overcome limitations of previous RL methods. The main contribution of the new system is to provide a flexible and generic model for multi-agent environments. The proposed generic model can be used in several applications, including competitive and cooperative problems, with the autonomous capacity to create fuzzy rules and expand their own rule structures, extracting knowledge from the direct interaction between the agents and the environment, without any use of supervised algorithms. The proposed model was tested in three different case studies, with promising results. The tests demonstrated that the developed system attained good capacity of convergence and coordination among the autonomous intelligent agents.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Benda, M., Jagannathan, V., & Dodhiawala, R. (1986). On optimal cooperation of knowledge sources an Empirical investigation. Technical report BCS-G2010-28, Boeing Advanced Technology Center, Boeing Computing Services, Seattle, Washington.

  2. Blaschke, J., Sebeke, C., & Rosenstiel, W. (2010). Organizing and planning the ASIC design process by means of a multi-agent system. In Proceedings of the international conference on agents and artificial intelligence (Vol. 1), Artificial Intelligence. INSTICC-2010, Valencia, Spain, pp. 22–24.

  3. Bodea, C. N., & Badea, I. R. (2010). Contributions to multi-agent systems implementation for project scheduling. In Proceedings of the 5th international conference on knowledge management: Projects, systems and technologies, Bucharest.

  4. Bodea, C.-N., Badea, I. R., & Purnus, A. (2010). Complex project scheduling using multi-agent methods: A case study for research projects. Management and Marketing, 3, 21–40.

    Google Scholar 

  5. Busoniu, L., Babuska, R., & De Schutter, B. A. (2008). Comprehensive survey of multiagent reinforcement learning. In systems, man, and cybernetics, part C: Applications and reviews. IEEE Transactions, 38(2), 156–172.

    Google Scholar 

  6. Claus, C., & Boutilier, C. (1998). The dynamics of reinforcement learning in cooperative multiagent systems. In Proceedings 15th national conference on artificial intelligence. 10th conference on artificial intelligence applications and innovation (AAAI/IAAI-98) (pp. 746–752). Madison, WI, July 26–30.

  7. Clouse, J. (1995). Learning from an automated training agent. Presented at the workshop agents that learn from other agents, 12th international conference on machine learning (ICML-95). Tahoe City, CA, July 9–12.

  8. Contreras, R., Vellasco, M., & Tanscheit, R. (2011). Hierarchical type-2 neuro-fuzzy BSP model. Information Sciences, 181(15), 3210–3224.

    Article  Google Scholar 

  9. Dosedla, M. (2009). Use of multi-agent systems in project management. In Proceedings of the 15th conference EEICT 2009. Vyd volume 4. Brno: VUT FEKT a FIT, 2009 (pp. 395–399). Brno: Brno University of Technology.

  10. Figueiredo, K., Vellasco, M., Pacheco, M. A. C., & Souza, F. J. (2004). Reinforcement learning-hierarchical neuro-fuzzy politree model for control of autonomous agents. In Fourth international conference on Hybrid Intelligent Systems (HIS’04) (pp. 130–135). Takamatsu, Japan: IEEE Computer Society.

  11. Figueiredo, K., Santos, M., Vellasco, M., & Pacheco, M. A. C. (2005). Modified reinforcement learning-hierarchical neuro-fuzzy politree model for control of autonomous agents. International Journal of Simulation Systems, Science and Technology, UK, 6(10/11), 4–13.

    Google Scholar 

  12. Fitch, R., Hengst, B., Suc, D., Calbert, G., & Scholz, J. B. (2005). Structural abstraction experiments in reinforcement learning. In Proceedings of the 18th Australian joint conference on artificial intelligence (AI-05), LNCS (Vol. 3809, pp. 164–175). Sydney, Australia, Dec. 5–9.

  13. Glorennec, P. Y., & Jouffe, L. (1997). Fuzzy Q-learning. In Proceedings of Fuzz-Ieee’97, sixth international conference on fuzzy systems, Barcelone, Espagne, Juillet (pp. 659–662).

  14. Gorodetsky, V., Karsaev, O., Konushy, V., Matzke, W.-E., Jentzsch, E., Ermolayev, V. (2006). Multi-agent software tool for management of design process in microelectronics. In Proceedings of intelligent agent technology (pp. 773–776).

  15. Jouffe, L. (1998). Fuzzy inference system learning by reinforcement methods. IEEE Transactions on Systems, Man and Cybernetics, Part C, 28(3), 338–355.

    Article  Google Scholar 

  16. Kaelbling, L. P., Littman, M. L., & Moore, W. A. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237–285.

    Google Scholar 

  17. Kok, J. R., Hoen, P .J’. T., Bakker, P. B., & Vlassis, N. (2005). Utile coordination: Learning interdependencies among cooperative agents. In Proceedings of the IEEE symposium on computational intelligence and games (CIG-05) (pp. 29–36). Colchester, UK, April 4–6.

  18. Linstone, H. A., & Turoff, M. (1975). The Delphi method: Techniques and applications. Reading, MA: Addison-Wesley.

    MATH  Google Scholar 

  19. Malcolm, D. G., Roseboom, J. H., Clark, C. E., & Fazar, W. (1959). Application of a technique for research and development program evaluation. Operations Research, 7(5), 646–669.

    Article  MATH  Google Scholar 

  20. Martins, F., Figueiredo, K., Vellasco, M., & Department of Electrical Engineering. (2010). Methods for acceleration of learning process of reinforcement learning neuro-fuzzy hierarchical politree model. International conference on autonomous and intelligent systems (AIS). ISBN: 978-1-4244-7104-1.

  21. Mcafee, R. P., & Mcmillan, J. (1987). Auctions and bidding. Journal of Economic Literature, 25, 699–738.

    Google Scholar 

  22. Nauck, D., & Kruse, R. (1997). Neuro-fuzzy systems for function approximation. In 4th international workshop fuzzy-neuro systems.

  23. Nokes, S. (2007). The definitive guide to project management (2nd ed.). London: Financial Times/Prentice Hall.

    Google Scholar 

  24. Price, B., & Boutilier, C. (2003). Accelerating reinforcement learning through implicit imitation. Journal of Artificial Intelligence Research, 19, 569–629.

    MATH  Google Scholar 

  25. Rabelo, E., Camarinha-Matos, L. M., & Afsarmanesh, H. (1999). Multi-agent-based agile scheduling. Robotics and Autonomous Systems, 27(1–2), 15–28.

    Article  Google Scholar 

  26. Ribeiro, C. H. C. (1999). A tutorial on reinforcement learning techniques. In International joint conference on neural networks. Washington: INNS Press.

  27. Sen, S., & Sekaran, M. (1995). Multiagent coordination with learning classifier systems In Proceedings of of the IJCAI workshop on adaptation and learning multiagent systems, Montreal (pp. 84–89).

  28. Shu-Guang, H., Er-Shi, Q., & Gang, L. (2005). A study on the project scheduling based on multi-agent systems. Mathematics in Practice and Theory, 1, 43–47.

    Google Scholar 

  29. Souza, F., Vellasco, M. M. B. R., & Pacheco, M. A. C. (2002). Hierarchical neuro-fuzzy QuadTree models. Fuzzy Sets and Systems, 130(2), 189–205.

    Article  MATH  MathSciNet  Google Scholar 

  30. STANDISH GROUP, CHAOS Summary. (2009). Online report. Accessed June 20, 2009 from http://www1.standishgroup.com/newsroom/chaos_2009.php.

  31. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.

    Google Scholar 

  32. Sutton, R. S. (1996). Advances in neural information processing systems 8. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Generalization in reinforcement learning: Successful examples using sparse coarse coding (pp. 1038–1044). Cambridge, MA: MIT Press.

    Google Scholar 

  33. Takeuchi, H., & Nonaka, I. (1986). The new product development game. Harvard business review. Accessed May 13, 2009 from http://hbr.org/product/new-new-product-development-game/an/86116-PDF-ENG.

  34. Tan, M. (1993). Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the 10th international conference on machine learning (ICML-93) (pp. 330–337). Amherst, OH, June 27–29.

  35. Vellasco, M. M. B. R., Pacheco, M. A. C., & Souza, F. J. (2004). Electric load forecasting: Evaluating the novel hierarchical neuro-fuzzy BSP model. Electrical Power and Energy Systems Elsevier Ltd, 26, 131–142.

    Article  Google Scholar 

  36. Walter, I., & Gomide, F. (2004). Evolução de Estratégias Nebulosas de Oferta em Mercados de Energia. In Congresso Brasileiro de Automática, Gramado. Anais do XV Congresso Brasileiro de Automática. Porto Alegre: SBA-UFRS.

  37. Weiss, G. (2000). Multiagent systems: A modern approach to distributed artificial intelligence (p. 648). Cambridge, MA: MIT Press.

    Google Scholar 

  38. Wu, C.-S., Chang, W.-C., Sethi, Ishwar, K. (2009). A metric-based multi-agent system for software project management. In Proceedings of the 8th IEEEACIS international conference on computer and information science ICIS (pp. 3–8).

  39. Yan, Y., Kuphal, T., & Bode, J. (2000). Application of multiagent systems in project management. International Journal of Production Economics, 68, 185–197.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rua Marquês de São Vicente, 225, Gávea, Rio de Janeiro, RJ, Brazil

    Marcelo França Corrêa, Marley Vellasco & Karla Figueiredo

  2. Núcleo de Computação Científica, CCMAT, Universidade Estadual da Zona Oeste (UEZO), Rio de Janeiro, Brazil

    Karla Figueiredo

Authors
  1. Marcelo França Corrêa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marley Vellasco
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Karla Figueiredo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marley Vellasco.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Corrêa, M.F., Vellasco, M. & Figueiredo, K. Multi-agent systems with reinforcement hierarchical neuro-fuzzy models. Auton Agent Multi-Agent Syst 28, 867–895 (2014). https://doi.org/10.1007/s10458-013-9242-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10458-013-9242-0

Keywords

Search

Navigation