Skip to main content
SpringerLink
Log in

Multi-agent Reinforcement Learning and Individuality Analysis for Cooperative Transportation with Obstacle Removal

  • Conference paper
  • First Online:
Distributed Autonomous Robotic Systems (DARS 2021)

Part of the book series: Springer Proceedings in Advanced Robotics ((SPAR,volume 22))

Included in the following conference series:

Abstract

Cooperative transportation is one of the essential tasks for multi-robot systems to imitate the decentralized systems of social insects. However, in a situation involving an obstacle on the pathway, multiple robots need to realize transportation and obstacle removal simultaneously. To address this multitasking problem, we first introduce a learning scenario and train robots’ decentralized policies via multi-agent reinforcement learning. Next, we propose two virtual experiments with blindfold teams and homogeneous teams to analyze the individual behaviors of the trained robots. The results showed that three robots with different policies performed two tasks simultaneously as a team. One robot’s policy tended to perform obstacle removal, and the other robots’ policies tended to perform cooperative transportation. Further, the first robot’s policy had the potential to perform two tasks simultaneously depending on the situation. Finally, we demonstrated the trained policies with three ground robots to show the feasibility of the system.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 219.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 279.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 279.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Beshers, S.N., Fewell, J.H.: Models of division of labor in social insects. Ann. Rev. Entomol. 46(1), 413–440 (2001)

    Article  Google Scholar 

  2. Robinson, G. E., Page, R. E.: Genetic basis for division of labor in an insect society. Genet. Soc. Evol. 61–80 (1989)

    Google Scholar 

  3. Nouyan, S., Gross, R., Bonani, M., Mondada, F., Dorigo, M.: Teamwork in self-organized robot colonies. IEEE Trans. Evol. Comput. 13, 695–711 (2009)

    Article  Google Scholar 

  4. Brutschy, A., Pini, G., Pinciroli, C., Birattari, M., Dorigo, M.: Self-organized task allocation to sequentially interdependent tasks in swarm robotics. Auton. Agent. Multi-Agent Syst. 28(1), 101–125 (2012). https://doi.org/10.1007/s10458-012-9212-y

    Article  Google Scholar 

  5. Garattoni, L., Birattari, M.: Autonomous task sequencing in a robot swarm. Sci. Robot. 3 (2018)

    Google Scholar 

  6. Bonabeau, E., Sobkowski, A., Theraulaz, G., Deneubourg, J.L.: Adaptive task allocation inspired by a model of division of labor in social insects. In: BCEC, pp. 36–45 (1997)

    Google Scholar 

  7. Ferrante, E., Turgut, A.E., Duéñez-Guzmán, E., Dorigo, M., Wenseleers, T.: Evolution of self-organized task specialization in robot swarms. PLoS Comput. Biol. 11(8), e1004273 (2015)

    Article  Google Scholar 

  8. Tuci, E., Alkilabi, M.H., Akanyeti, O.: Cooperative object transport in multi-robot systems: a review of the state-of-the-art. Front. Robot. AI 5, 59 (2018)

    Article  Google Scholar 

  9. Kosuge, K., Oosumi, T.: Decentralized control of multiple robots handling an object. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 1996, vol. 1, pp. 318–323. IEEE (1996)

    Google Scholar 

  10. Wang, Z., Schwager, M.: Kinematic multi-robot manipulation with no communication using force feedback. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 427-432. IEEE, May 2016

    Google Scholar 

  11. Wang, Z., Yang, G., Su, X., Schwager, M.: OuijaBots: omnidirectional robots for cooperative object transport with rotation control using no communication. In: Groß, R., et al. (eds.) Distributed Autonomous Robotic Systems. SPAR, vol. 6, pp. 117–131. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73008-0_9

    Chapter  Google Scholar 

  12. Culbertson, P., Schwager, M.: Decentralized adaptive control for collaborative manipulation. In: 2018 IEEE International Conference on Robotics and Automation, pp. 278–285. IEEE (2018)

    Google Scholar 

  13. Farivarnejad, H., Wilson, S., Berman, S.: Decentralized sliding mode control for autonomous collective transport by multi-robot systems. In: 2016 IEEE 55th Conference on Decision and Control, pp. 1826–1833. IEEE (2016)

    Google Scholar 

  14. Chen, J., Gauci, M., Li, W., Kolling, A., Gross, R.: Occlusion-based cooperative transport with a swarm of miniature mobile robots. IEEE Trans. Rob. 31(2), 307–321 (2015)

    Article  Google Scholar 

  15. Gross, R., Dorigo, M.: Towards group transport by swarms of robots. Int. J. Bio-Inspir. Comput. 1(1–2), 1–13 (2009)

    Article  Google Scholar 

  16. Alkilabi, M.H.M., Narayan, A., Tuci, E.: Cooperative object transport with a swarm of e-puck robots: robustness and scalability of evolved collective strategies. Swarm Intell. 11(3–4), 185–209 (2017)

    Article  Google Scholar 

  17. Rahimi, M., Gibb, S., Shen, Y., La, H.M.: A comparison of various approaches to reinforcement learning algorithms for multi-robot box pushing. In: Fujita, H., Nguyen, D.C., Vu, N.P., Banh, T.L., Puta, H.H. (eds.) ICERA 2018. LNNS, vol. 63, pp. 16–30. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-04792-4_6

    Chapter  Google Scholar 

  18. Zhang, L., Sun, Y., Barth, A., Ma, O.: Decentralized control of multi-robot system in cooperative object transportation using deep reinforcement learning. IEEE Access 8, 184109–184119 (2020)

    Article  Google Scholar 

  19. Wang, Y., de Silva, C.W.: Cooperative transportation by multiple robots with machine learning. In: 2006 IEEE International Conference on Evolutionary Computation, pp. 3050–3056. IEEE (2006)

    Google Scholar 

  20. Gupta, J.K., Egorov, M., Kochenderfer, M.: Cooperative multi-agent control using deep reinforcement learning. In: Sukthankar, G., Rodriguez-Aguilar, J.A. (eds.) AAMAS 2017. LNCS (LNAI), vol. 10642, pp. 66–83. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-71682-4_5

    Chapter  Google Scholar 

  21. Hernandez-Leal, P., Kartal, B., Taylor, M.E.: Agent modeling as auxiliary task for deep reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 15, no. 1, pp. 31–37 (2019)

    Google Scholar 

  22. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. Adv. Neural. Inf. Process. Syst. 30, 6379–6390 (2017)

    Google Scholar 

  23. Sutton, R. S., McAllester, D. A., Singh, S. P., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. Adv. Neural Inf. Process. Syst. 1057–1063 (2000)

    Google Scholar 

  24. Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41–48 (2009)

    Google Scholar 

  25. Cobbe, K., Klimov, O., Hesse, C., Kim, T., Schulman, J.: Quantifying generalization in reinforcement learning. In: International Conference on Machine Learning, pp. 1282–1289. PMLR (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Data Analytics Research Domain, Toyota Central R&D Labs., Inc., 41-1 Yokomichi, Nagakute, Aichi, Japan

    Takahiro Niwa, Kazuki Shibata & Tomohiko Jimbo

Authors
  1. Takahiro Niwa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kazuki Shibata
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tomohiko Jimbo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Takahiro Niwa .

Editor information

Editors and Affiliations

  1. Mechanical Engineering and Science, Kyoto University, Kyoto, Japan

    Fumitoshi Matsuno

  2. Mechanical Systems Engineering, Nagoya University, Nagoya, Japan

    Shun-ichi Azuma

  3. Computer Science and Information Technology, Hokkaido University, Sapporo, Hokkaido, Japan

    Masahito Yamamoto

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Niwa, T., Shibata, K., Jimbo, T. (2022). Multi-agent Reinforcement Learning and Individuality Analysis for Cooperative Transportation with Obstacle Removal. In: Matsuno, F., Azuma, Si., Yamamoto, M. (eds) Distributed Autonomous Robotic Systems. DARS 2021. Springer Proceedings in Advanced Robotics, vol 22. Springer, Cham. https://doi.org/10.1007/978-3-030-92790-5_16

Download citation

Publish with us

Policies and ethics

Search

Navigation