Skip to main content
SpringerLink
Log in

6D Localization and Kicking for Humanoid Robotic Soccer

  • Regular Paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Robotic soccer simulation is a challenging area, where the development of new techniques is paramount to remain competitive. Robotic skill evolution has accelerated with recent developments in deep learning algorithms, leading to improvements in behavior number and complexity. Shooting a ball towards a defined target is one of the most basic yet indispensable skills in soccer. However, fast and accurate kicks pose several challenges. In order to reach that target, the skill is highly dependent on the ability of the agent to self-locate and self-orient, in order to better position itself before the kick. To tackle these issues, a 6D localization technique was devised. To optimize the kick behavior, two scenarios were proposed. In the first, the robot walks to the ball, stops, and then kicks. In the second, it kicks the ball while moving. We used state-of-the-art algorithms — Proximal Policy Optimization and Soft Actor Critic — to solve these complex problems and show their applicability in the context of RoboCup. Obtained results have shown very significant improvements over previously used behaviors by FC Portugal 3D team. The new kick in motion executes 5 times faster than the previous kick, and the new 6D pose estimator has an average error of just 6.3mm, a reduction of more than 97%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: Large-scale machine learning on heterogeneous systems. https://www.tensorflow.org/(2015)

  2. Abdolmaleki, A., Simões, D., Lau, N., Reis, L. P., Neumann, G., Sarıel, S., Lee, D.D.: Learning a Humanoid Kick with Controlled Distance. In: Behnke, S., Sheh, R. (eds.) Robocup 2016: Robot world cup XX, 45–57. Springer International Publishing, Cham (2017)

  3. Abreu, M., Reis, L.P., Lau, N.: Learning to Run Faster in a Humanoid Robot Soccer Environment through Reinforcement Learning. In: Chalup, S., Niemueller, T., Suthakorn, J., Williams, M.A. (eds.) Robocup 2019: Robot World Cup XXIII, 3-15. Springer International Publishing, Cham (2019)

  4. Boedecker, J., Asada, M.: Simspark–concepts and application in the robocup 3d soccer simulation league SIMPAR-2008 Workshop on The Universe of RoboCup Simulators (2008)

  5. Brafman, R. I., Tennenholtz, M.: R-MAX – a general polynomial time algorithm for near-optimal reinforcement learning. J. Mach. Learn. Res. 3(Oct), 213–231 (2002)

    MathSciNet  MATH  Google Scholar 

  6. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym (2016)

  7. Bustamante, C.: Probabilistic agent localization and Fuzzy-Bayesian pass evaluation for the RoboCup simulation 3D league. Master’s Thesis, Tecnológico De Monterrey, Monterrey, Mexico (2007)

  8. Coulom, R.: Clop: Confident Local Optimization for Noisy Black-Box Parameter Tuning. In: Advances in Computer Games, pp. 146–157. Springer (2011)

  9. Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., Wu, Y., Zhokhov, P.: Openai baselines https://github.com/openai/baselines (2017)

  10. Documentation, A.: NAO - actuator & sensor list. http://doc.aldebaran.com/2-1/family/nao_dcm/actuator_sensor_names.html

  11. Dorer, K.: Learning to Use Toes in a Humanoid Robot. In: Akiyama, H., Obst, O., Sammut, C., Tonidandel, F. (eds.) Robocup 2017: Robot World Cup XXI, 168–179. Springer International Publishing, Cham (2018)

  12. Ferreira, R., Reis, L.P., Moreira, A.P., Lau, N.: Development of an Omnidirectional Kick for a NAO Humanoid Robot. In: Pavón, J., Duque-Méndez, N.D., Fuentes-Fernández, R. (eds.) Advances in Artificial Intelligence – IBERAMIA 2012, 571–580. Springer Berlin Heidelberg, Berlin, Heidelberg (2012)

  13. Golub, G.H., Van Loan, C.F.: An analysis of the total least squares problem. SIAM J. Num. Anal. 17(6), 883–893 (1980)

    Article  MathSciNet  Google Scholar 

  14. Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 80, pp. 1861–1870. PMLR, Stockholmsmässan, Stockholm Sweden (2018)

  15. Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., Levine, S.: Soft actor-critic algorithms and applications. arXiv:1812.05905 (2018)

  16. Hao, Y., Liang, Z., Liu, J., Li, J., Zhao, H.: The Framework Design of Humanoid Robots in the RoboCup 3D Soccer Simulation Competition. In: 2013 10Th IEEE International conference on control and automation (ICCA), pp. 1423–1428. IEEE (2013)

  17. Hester, T., Quinlan, M., Stone, P.: Generalized Model Learning for Reinforcement Learning on a Humanoid Robot. In: 2010 IEEE International Conference on Robotics and Automation, pp. 2369–2374. IEEE (2010)

  18. Hester, T., Stone, P.: Negative Information and Line Observations for Monte Carlo Localization. In: 2008 IEEE International Conference on Robotics and Automation, pp. 2764–2769. IEEE (2008)

  19. Hester, T., Stone, P.: Generalized model learning for reinforcement learning in factored domains. In: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 2, pp. 717–724 (2009)

  20. Hill, A., Raffin, A., Ernestus, M., Gleave, A., Kanervisto, A., Traore, R., Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., Wu, Y.: Stable baselines https://github.com/hill-a/stable-baselines (2018)

  21. Hoffman, J., Spranger, M., Gohring, D., Jungel, M.: Making Use of What You Don’t See: Negative Information in Markov Localization. In: 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2947–2952. IEEE (2005)

  22. Jakob, W., Rhinelander, J., Moldovan, D.: pybind11 – seamless operability between c++ 11 and python. https://github.com/pybind/pybind11 (2017)

  23. Jouandeau, N., Hugel, V.: Optimization of Parametrised Kicking Motion for Humanoid Soccer Player. In: 2014 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pp. 241–246. IEEE (2014)

  24. Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I., Osawa, E., Matsubara, H.: RoboCup: A challenge problem for AI. AI Magazine 18(1), 73–85 (1997). https://doi.org/10.1609/aimag.v18i1.1276

    Google Scholar 

  25. Lu, W., Zhang, J., Zhao, X., Wang, J., Dang, J.: Multimodal sensory fusion for soccer robot self-localization based on long short-term memory recurrent neural network. J. Ambient. Intell. Humaniz. Comput. 8(6), 885–893 (2017)

    Article  Google Scholar 

  26. MacAlpine, P., Depinet, M., Liang, J., Stone, P.: UT Austin Villa: RoboCup 2014 3D simulation league competition and technical challenge champions. In: Bianchi, R.A.C., Akin, H.L., Ramamoorthy, S., Sugiura, K. (eds.) RoboCup-2014: Robot Soccer World Cup XVIII, Lecture Notes in Artificial Intelligence. Springer (2015)

  27. MacAlpine, P., Stone, P.: UT Austin Villa: RoboCup 2017 3D simulation league competition and technical challenges champions. In: Sammut, C., Obst, O., Tonidandel, F., Akyama, H. (eds.) RoboCup 2017: Robot soccer world cup XXI. Springer (2017)

  28. Müller, J., Laue, T., Röfer, T.: Kicking a ball - modeling complex dynamic motions for humanoid robots. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 6556 LNAI, pp 109–120 (2011)

  29. Muzio, A., Aguiar, L., Máximo, M.R., Pinto, S.C.: Monte Carlo Localization with Field Lines Observations for Simulated Humanoid Robotic Soccer. In: 2016 XIII Latin American Robotics Symposium and IV Brazilian Robotics Symposium (LARS/SBR), pp. 334–339. IEEE (2016)

  30. Omidvar, M.N., Li, X.: A Comparative Study of Cma-Es on Large Scale Global Optimisation. In: Australasian Joint Conference on Artificial Intelligence, pp. 303–312. Springer (2010)

  31. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: an Imperative Style, High-Performance Deep Learning Library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc (2019)

  32. Raffin, A., Hill, A., Ernestus, M., Gleave, A., Kanervisto, A., Dormann, N.: Stable baselines3 https://github.com/DLR-RM/stable-baselines3 (2019)

  33. RoboCup: 3D Soccer Simulation League history. https://ssim.robocup.org/3d-simulation/3d-history/ (2021)

  34. Robotics, S.: NAO the humanoid robot. https://www.softbankrobotics.com/emea/en/nao

  35. Shafii, N., Abdolmaleki, A., Ferreira, R., Lau, N., Reis, L.P.: Omnidirectional Walking and Active Balance for Soccer Humanoid Robot. In: Correia, L., Reis, L.P., Cascalho, J. (eds.) Progress in Artificial Intelligence, pp. 283–294. Springer Berlin Heidelberg, Berlin, Heidelberg (2013)

  36. Shafii, N., Lau, N., Reis, L.P.: Learning to walk fast: Optimized hip height movement for simulated and real humanoid robots. J. Intel. & Robotic Syst. 80(3), 555–571 (2015). https://doi.org/10.1007/s10846-015-0191-5

    Article  Google Scholar 

  37. SimSpark: Agentsyncmode — SimSpark. http://simspark.sourceforge.net/wiki/index.php/AgentSyncMode (2012)

  38. Snafii, N., Abdolmaleki, A., Lau, N., Reis, L.P.: Development of an omnidirectional walk engine for soccer humanoid robots. Inter. J. Advanced Robotic Syst. 12(12), 193 (2015). https://doi.org/10.5772/61314

    Google Scholar 

  39. Söderkvist, I.: Using SVD for some fitting problems. https://ltu.se/cms_fs/1.51590!/svd-fitting.pdf (2009). Luleå University of Technology

  40. Stoecker, J., Visser, U.: Roboviz: Programmable visualization for simulated soccer, P. 282–293. Springer-Verlag, Berlin, Heidelberg (2012)

    Google Scholar 

  41. Sutton, R.S., Barto, A.G.: Reinforcement learning: An Introduction, second edn The MIT Press (2018)

  42. Teixeira, H., Silva, T., Abreu, M., Reis, L.P.: Humanoid robot kick in motion ability for playing robotic soccer. In: 2020 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pp. 34–39. https://doi.org/10.1109/ICARSC49921.2020.9096073 (2020)

  43. Thrun, S., Burgard, W., Fox, D.: Probabilistic Robotics (Intelligent Robotics and Autonomous Agents). The MIT Press, Cambridge, Massachusetts (2005)

    MATH  Google Scholar 

  44. Visser, U., Burkhard, H.D.: Robocup: 10 Years of Achievements and Future Challenges. AI Mag. 28(2), 115–132 (2007)

    Google Scholar 

  45. Watkins, C.: Learning from delayed rewards. Ph.D. thesis, University of Cambridge (1989)

  46. Xu, Y., Vatankhah, H.: Simspark: an open source robot simulator developed by the Robocup community. In: Behnke, S., Veloso, M., Visser, A., Xiong, R. (eds.) Robocup 2013: Robot World Cup XVII, pp. 632–639. Springer, Berlin, Heidelberg (2014)

Download references

Funding

The first author is supported by FCT — Foundation for Science and Technology under grant SFRH/BD/139926/2018. The work was also partially funded by COMPETE 2020 and FCT, under projects UIDB/00027/2020 (LIACC) and UIDB/00127/2020 (IEETA).

Author information

Author notes

    Authors and Affiliations

    1. FEUP - Faculty of Engineering of the University of Porto, Porto, Portugal

      Miguel Abreu, Tiago Silva, Henrique Teixeira & Luís Paulo Reis

    2. LIACC - Artificial Intelligence and Computer Science Laboratory, University of Porto, Porto, Portugal

      Miguel Abreu, Tiago Silva & Luís Paulo Reis

    3. IEETA - Institute of Electronics and Informatics Engineering of Aveiro, University of Aveiro, Aveiro, Portugal

      Nuno Lau

    Authors
    1. Miguel Abreu
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Tiago Silva
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Henrique Teixeira
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Luís Paulo Reis
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Nuno Lau
      View author publications

      You can also search for this author in PubMed Google Scholar

    Contributions

    All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by all authors. The first draft of the manuscript was written by Miguel Abreu, Tiago Silva, and Henrique Teixeira. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

    Corresponding author

    Correspondence to Miguel Abreu.

    Ethics declarations

    Conflict of Interests

    The authors declare that they have no conflict of interest.

    Additional information

    Publisher’s Note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Miguel Abreu and Tiago Silva contributed equally to this work as first authors.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Abreu, M., Silva, T., Teixeira, H. et al. 6D Localization and Kicking for Humanoid Robotic Soccer. J Intell Robot Syst 102, 30 (2021). https://doi.org/10.1007/s10846-021-01385-3

    Download citation

    • Received:

    • Accepted:

    • Published:

    • DOI: https://doi.org/10.1007/s10846-021-01385-3

    Keywords

    Mathematics Subject Classification (2010)

    Search

    Navigation