Skip to main content

Advertisement

SpringerLink
Log in

Generation a shooting on the walking for soccer simulation 3D league using Q-learning algorithm

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

RoboCup is one of the greatest human endeavors to operationalize research in the field of robotics and artificial intelligence. RoboCup soccer simulation 3D (RoboCup3D) competition provide a great opportunity to work with humanoid robots without the need for hardware. One of the most important goals of the teams participating in RoboCup3D league is the ability to increase the number of shoots. The reason for this importance is that superiority over the opponent requires a powerful and precise shoot. The methods introduced for shooting so far are mostly based on Inverse Kinematics (IK) and point analysis. The assumption of these methods is that the positions of the robot and the ball is fixed. However, this is not always the case for shooting. In this paper, a shooting strategy is presented for situations where the robot is walking. Here, a curved path is designed to movement the robot towards the ball so that the robot will eventually have an optimal position to shoot. In general, the vision preceptor in RoboCup3D has noise. Hence, robot movement parameters such as speed and angle are more precisely adjusted by the Q-learning algorithm. Finally, when the robot is in the optimal position relative to the ball and the goal, the IK module is applied to the shooting strategy. Simulations and experiments prove the superiority of the proposed algorithm compared to most teams in RoboCupSoccer and Iran’s Open RoboCup3D leagues.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Abreu M, Reis LP, Lau N (2019) Learning to run faster in a humanoid robot soccer environment through reinforcement learning. In Robot World Cup, pp. 3–15

  • Asghari P, Soleimani E, Nazerfard E (2020) Online human activity recognition employing hierarchical hidden Markov models. J Ambient Intell Humaniz Comput 11(3):1141–1152

    Article  Google Scholar 

  • Aslanpour MS, Dashti SE, Ghobaei-Arani M, Rahmanian AA (2018) Resource provisioning for cloud applications: a 3-D, provident and flexible approach. J Supercomput 74(12):6470–6501

    Article  Google Scholar 

  • Böckmann A, Laue T (2016) Kick motions for the NAO robot using dynamic movement primitives. In: Robot World Cup, pp. 33–44

  • Boedecker J, Asada M (2008) Simspark–concepts and application in the robocup 3D soccer simulation league. In: Proc workshop SIMPAR int conf simulation, modeling programming autonomous robots, pp. 174–181

  • Dorer K, Glaser S (2016) The magmaoffenburg 2016 robocup 3d simulation team. Proceedings of RoboCup, pp. 1–6

  • Farazi H, Allgeuer P, Ficht G, Brandenburger A, Pavlichenko D, Schreiber M, Behnke S (2016) RoboCup 2016 humanoid teensize winner nimbro: robust visual perception and soccer behaviors. In: Robot World Cup, pp. 478–490

  • Forouzandeh S, Rostami M, Berahmand K (2021) Presentation a Trust Walker for rating prediction in recommender system with biased random walk: effects of H-index centrality, similarity in items and friends. Eng Appl Artif Intell 104:104325

    Article  Google Scholar 

  • Gerndt R, Seifert D, Baltes JH, Sadeghnejad S, Behnke S (2015) Humanoid robots in soccer: robots versus humans in RoboCup 2050. IEEE Robot Autom Mag 22(3):147–154

    Article  Google Scholar 

  • Ghobaei-Arani M, Rahmanian AA, Souri A, Rahmani AM (2018) A moth-flame optimization algorithm for web service composition in cloud computing: simulation and verification. Softw Pract Exp 48(10):1865–1892

    Google Scholar 

  • Han SC, Li HX (2004) Invertible incline matrices and Cramer’s rule over inclines. Linear Algebr Appl 389(2004):121–138

    Article  MathSciNet  MATH  Google Scholar 

  • He H, Liang Z, Lu Y, Xu C, Yang B, Fang F (2019) Dynamic kick optimization of humanoid robot based on options framework. In 2019 Chinese control and decision conference (CCDC), pp. 5176–5181

  • Hereid A, Cousineau EA, Hubicki CM, Ames AD (2016) 3D dynamic walking with underactuated humanoid robots: a direct collocation framework for optimizing hybrid zero dynamics. In 2016 IEEE int conf on robotics and automation (ICRA), pp. 1447–1454

  • Huang Q, Yokoi K, Kajita S, Kaneko K, Arai H, Koyachi N, Tanie K (2001) Planning walking patterns for a biped robot. IEEE Trans Robot Autom 17(3):280–289

    Article  Google Scholar 

  • Kim D, Lee T, Kim S, Lee B, Youn HY (2020) Adaptive packet scheduling in iot environment based on q-learning. J Ambient Intell Humaniz Comput 11(6):2225–2235

    Article  Google Scholar 

  • Kober J, Oztop E, Peters J (2011) Reinforcement learning to adjust robot movements to new situations. In twenty-second int joint conf on artificial intelligence, pp. 2650–2655

  • Li X, Liang Z, Feng H (2015) Kicking motion planning of NAO robots based on CMA-ES. In The 27th Chinese control and decision conf. (2015 CCDC), pp. 6158–6161

  • Li Z, Zhao T, Chen F, Hu Y, Su CY, Fukuda T (2017) Reinforcement learning of manipulation and grasping using dynamical movement primitives for a humanoid like mobile manipulator. IEEE/ASME Trans Mechatron 23(1):121–131

    Article  Google Scholar 

  • MacAlpine P, Stone P (2016) UT Austin villa: RoboCup 2016 3D simulation league competition and technical challenges champions. In Robot World Cup, pp. 515–528

  • MacAlpine P, Depinet M, Liang J, Stone P (2014) UT Austin villa: RoboCup 2014 3D simulation league competition and technical challenge champions. In Robot Soccer World Cup, pp. 33–46

  • Mahapatra A, Roy SS, Pratihar DK (2013) Modeling and simulation of wave gait of a hexapod walking robot: A CAD/CAE approach. IAES Int J Robot Autom 2(3):104–111

    Google Scholar 

  • Marques H, Lau N, Reis LP (2004) FC Portugal 3D simulation team: architecture, low-level skills and team behaviour optimized for the new robocup 3D simulator. In: Proc scientific meeting of the Portuguese robotics open, pp. 31–37

  • Nasrollahi P, Jafari S, Ebrahimi M (2012) Action classification of humanoid soccer robots using machine learning. In: The 16th CSI international symposium on artificial intelligence and signal processing (AISP 2012), pp. 598–603

  • Pena P, Masterjohn J, Visser U (2017) An omni-directional kick engine for humanoid robots with parameter optimization. In: Robot World Cup, pp. 385–397

  • Qian Y, Baucom A, Han Q, Small A, Buckman D, Lee DD (2016) The UPennalizers RoboCup standard platform league team description paper 2016. In RoboCup symposium

  • Rakojević B, Mrdaković V, Pažin N, Vulović R, Leontijević B, Ilić D (2019) Speed-accuracy tradeoff of instep kick in young soccer players. Facta Univ Ser Phys Educ Sport 16(3):543–555

    Google Scholar 

  • Rezaeipanah A, Amiri P, Jafari S (2020) Performing the kick during walking for robocup 3d soccer simulation league using reinforcement learning algorithm. Int J Soc Robot 13(2020):1235–1252

    Google Scholar 

  • Rezaeipanah A, Jamshidi Z, Jafari S (2021) A shooting strategy when moving on humanoid robots using inverse kinematics and q-learning. Int J Robot Autom 36(3):1–7

    Google Scholar 

  • Shawki N, Lazarou G, Isenberg DR (2020) Stability and performance analysis of a payload-manipulating robot without adaptive control. Int J Robot Autom 35(1):15–21

    Google Scholar 

  • Stoecker J, Visser U (2013) Visualizing and debugging complex multi-agent soccer scenes in real time. In: Robot Soccer World Cup. Springer, Berlin, pp. 640–647

  • Stone P, Sutton RS, Kuhlmann G (2005) Reinforcement learning for robocup soccer keepaway. Adapt Behav 13(3):165–188

    Article  Google Scholar 

  • Tillaar R, Fuglstad P (2017) Effect of instructions prioritizing speed or accuracy on kinematics and kicking performance in football players. J Mot Behav 49(4):414–421

    Article  Google Scholar 

  • Wang C, Jia W, Sun Y, Ma S (2019) Practical kicking motion generation method for NAO. In 2019 IEEE int conf on mechatronics and automation (ICMA), pp. 163–168

  • Yasin A, Huang Q, Xu Q, Sultan MS (2012) Humanoid robot push recovery through foot placement. In: 2012 IEEE int conf on mechatronics and automation, pp. 59–63

Download references

Acknowledgements

2020 Yantai school land integration development project: Artificial intelligence specialty and comprehensive training platform based on new generation information technology. (2020XDRHXMXK09)

Author information

Authors and Affiliations

  1. School of Information Engineering, Yantai Institute of Technology, Yantai, Shandong, 264005, China

    Yun Li & Yibin Song

  2. Department of Computer Engineering, University of Rahjuyan Danesh Borazjan, Bandar Bushehr, Iran

    Amin Rezaeipanah

Authors
  1. Yun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yibin Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amin Rezaeipanah
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yibin Song.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Song, Y. & Rezaeipanah, A. Generation a shooting on the walking for soccer simulation 3D league using Q-learning algorithm. J Ambient Intell Human Comput 14, 6947–6957 (2023). https://doi.org/10.1007/s12652-021-03551-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-021-03551-9

Keywords

Search

Navigation