Skip to main content

Advertisement

Springer Nature Link
Log in

Neuroevolution of Inverted Pendulum Control: A Comparative Study of Simulation Techniques

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

The inverted pendulum control problem is a classical benchmark in control theory. Amongst the approaches to developing control programs for an inverted pendulum, the evolution of Artificial Neural Network (ANN) based controllers has received some attention. The authors have previously shown that Evolutionary Robotics (ER) can successfully be used to evolve inverted pendulum stabilization controllers in simulation and that these controllers can transfer successfully from simulation to real-world robotic hardware. During this process, use was made of robotic simulators constructed from empirically-collected data and based on ANNs. The current work aims to compare this method of simulator construction with the more traditional method of building robotic simulators based on physics equations governing the robotic system under consideration. In order to compare ANN-based and physics-based simulators in the evolution of inverted pendulum controllers, a real-world wheeled inverted pendulum robot was considered. Simulators based on ANNs as well as on a system of ordinary differential equations describing the dynamics of the robot were developed. These two simulation techniques were then compared by using each in the simulation-based evolution of controllers. During the evolution process, the effects of injecting different levels of noise into the simulation was furthermore studied. Encouraging results were obtained, with controllers evolved using ANN-based simulators and realistic levels of noise outperforming those evolved using the physics-based simulators.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Encog Machine Learning Framework — Heaton Research (Accessed January 2015). http://www.heatonresearch.com/encog

  2. NXT Acceleration / Tilt Sensor (Accessed July 2016). https://www.hitechnic.com/cgi-bin/commerce.cgi?preadd=action&key=NAC1040

  3. NXT Gyro Sensor (Accessed July 2016). https://www.hitechnic.com/cgi-bin/commerce.cgi?preadd=action&key=NGY1044

  4. nxtOSEK/JSP: ANSI C/C++ with OSEK/ μITRON RTOS for Lego MindStorms NXT (Accessed November 2014). http://lejos-osek.sourceforge.net/

  5. Commons Math: The Apache Commons Mathematics Library (Accessed September 2014). http://commons.apache.org/proper/commons-math/

  6. Atkeson, C.G., Santamaria, J.C.: A Comparison of Direct and Model-Based Reinforcement Learning. In: International Conference on Robotics and Automation, pp. 3557–3564. IEEE Press (1997)

  7. Bonarini, A., Caccia, C., Lazaric, A., Restelli, M.: Batch Reinforcement Learning for Controlling a Mobile Wheeled Pendulum Robot. In: Artificial Intelligence in Theory and Practice II, pp. 151–160. Springer (2008)

  8. Bongard, J., Zykov, V., Lipson, H.: Resilient machines through continuous self-modeling. Science 314, 1118–1121 (2006)

    Article  Google Scholar 

  9. Burden, R., Faires, J.: Numerical Analysis, 8th Edn. Thomson Brooks/Cole, London (2005)

    MATH  Google Scholar 

  10. Canale, M., Brunet, S.C.: A Lego Mindstorms NXT Experiment for Model Predictive Control Education. In: European Control Conference (ECC) (2013)

  11. Colton, S.: The balance filter: A simple solution for integrating accelerometer and gyroscope measurements for a balancing platform. White paper Massachusetts Institute of Technology (2007)

  12. De Nardi, R., Holland, O.E.: Coevolutionary modelling of a miniature rotorcraft. Intelligent Autonomous Systems 10 (2008). IAS-10

  13. Drumwright, E., Hsu, J., Koenig, N., Shell, D.: Extending Open Dynamics Engine for robotics simulation. Simulation, Modeling, and Programming for Autonomous Robots 6472, 38–50 (2010)

    Article  Google Scholar 

  14. El-Hawwary, M.I., Elshafei, A.L., Emara, H.M., Fattah, H.A.A.: Adaptive fuzzy control of the inverted pendulum problem. IEEE Trans. Control Syst. Technol. 14(6), 1135–1144 (2006)

    Article  Google Scholar 

  15. Engelbrecht, A.P.: Computational Intelligence: An Introduction, 2nd edn. Wiley, West Sussex (2007)

    Book  Google Scholar 

  16. Floreano, D., Mondada, F.: Evolution of homing navigation in a real mobile robot. IEEE Trans. Syst. Man Cybern. 26(3), 396–407 (1996)

  17. Glette, K., Klaus, G., Zagal, J.C., Torresen, J.: Evolution of Locomotion in a Simulated Quadruped Robot and Transferral to Reality. In: Proceedings of the Seventeenth International Symposium on Artificial Life and Robotics (2012)

  18. Gomez, F.J., Miikkulainen, R.: Transfer of Neuroevolved Controllers in Unstable Domains. In: Proceedings of the Genetic Evolutionary Computation Conference (GECCO-04) (2004)

  19. Gürocak, H.: A genetic-algorithm-based method for tuning fuzzy logic controllers. Fuzzy Set. Syst. 108(1), 39–47 (1999)

  20. Hapke, M., Komosinski, M.: Evolutionary design of interpretable fuzzy controllers. Foundation of Computing and Decision Sciences 33(4), 351 (2008)

  21. Hartland, C., Bredeche, N.: Evolutionary Robotics, Anticipation and the Reality Gap. In: IEEE International Conference on Robotics and Biomimetics (2006)

  22. Heidrich-Meisner, V., Igel, C.: Neuroevolution strategies for episodic Reinforcement Learning. J. Algoritm. 64(4), 152–168 (2009)

    Article  MATH  Google Scholar 

  23. Herrera, F., Lozano, M., Verdegay, J.: Tuning fuzzy logic controllers by genetic algorithms. Int. J. Approx. Reason. 12(3), 299–315 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  24. Holland, J.H.: Adaptation in Natural and Artificial Systems. MIT Press, Cambridge (1992)

    Google Scholar 

  25. Hotz, P., Gómez, G.: The Transfer Problem from Simulation to the Real World in Artificial Evolution. In: Workshop and Tutorial Proceedings of the Ninth International Conference on the Simulation and Synthesis of Living Systems (Alife IX) (2004)

  26. Huang, C.H., Wang, W.J., Chiu, C.H.: Design and implementation of fuzzy control on a two-wheel inverted pendulum. IEEE Trans. Ind. Electron. 58(7), 2988–3001 (2011)

    Article  Google Scholar 

  27. Igel, C.: Neuroevolution for Reinforcement Learning Using Evolution Strategies. In: Congress on Evolutionary Computation (CEC), vol. 4, pp. 2588–2595 (2003)

  28. Jakobi, N.: Minimal simulations for evolutionary robotics. Ph.D. thesis University of Sussex (1998)

  29. Jakobi, N.: Running across the Reality Gap: Octopod Locomotion Evolved in a Minimal Simulation. In: Evolutionary Robotics, pp. 39–58. Springer (1998)

  30. Jakobi, N., Husbands, P., Harvey, I.: Noise and the Reality Gap: The Use of Simulation in Evolutionary Robotics, vol. 929. Springer, Berlin (1995)

    Google Scholar 

  31. Kawada, K., Yamamoto, T., Mada, Y.: A Design of Evolutionary Recurrent Neural-Net Based Controllers for an Inverted Pendulum. In: Fifth Asian Control Conference (2004)

  32. Klaus, G., Glette, K., Tørresen, J.: A Comparison of Sampling Strategies for Parameter Estimation of a Robot Simulator. In: Proceedings of the Third International Conference on Simulation, Modeling, and Programming for Autonomous Robots (2012)

  33. Koos, S., Cully, A., Mouret, J.: Fast damage recovery in robotics with the T-Resilience algorithm. Preprint version (available: arXiv:1302.0386,2013)

  34. Koos, S., Mouret, J., Doncieux, S.: The Transferability Approach: Crossing the reality gap in Evolutionary Robotics. IEEE Trans. Evol. Comput., 1–25 (2012)

  35. Li, Z., Xu, C.: Adaptive fuzzy logic control of dynamic balance and motion for wheeled inverted pendulums. Fuzzy Set. Syst. 160(12), 1787–1803 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  36. Lipson, H., Bongard, J., Zykov, V., Malone, E.: Evolutionary Robotics for Legged Machines: From Simulation to Physical Reality. In: Proceedings of the 9th International Conference on Intelligent Autonomous Systems (2006)

  37. Moeckel, R., Perov, Y.N., Nguyen, A.T., Vespignani, M., Bonardi, S., Pouya, S., Sproewitz, A., van den Kieboom, J., Wilhelm, F., Ijspeert, A.J.: Gait Optimization for Roombots Modular Robots - Matching Simulation and Reality. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (2013)

  38. Nakamura, S., Hashimoto, S.: Hybrid Learning Strategy to Solve Pendulum Swing-Up Problem for Real Hardware. In: IEEE International Conference on Robotics and Biomimetics (2007)

  39. Nelles, O.: Nonlinear System Identification: From Classical Approaches to Neural Networks and Fuzzy Models. Springer, Berlin (2010)

    MATH  Google Scholar 

  40. Nelson, A.L., Barlow, G.J., Doitsidis, L.: Fitness functions in evolutionary robotics: A survey and analysis. Robot. Auton. Syst. 57(4), 345–370 (2009)

  41. Nguyen-Tuong, D., Peters, J.: Model learning for robot control: A survey. Cogn. Process. 12(4), 319–340 (2011)

    Article  Google Scholar 

  42. Oh, S.K., Pedrycz, W., Rho, S.B., Ahn, T.C.: Parameter estimation of fuzzy controller and its application to inverted pendulum. Eng. Appl. Artif. Intell. 17(1), 37–60 (2004)

    Article  Google Scholar 

  43. Pratihar, D.K.: Evolutionary Robotics - A review. Sadhana 28(6), 999–1009 (2003)

    Article  Google Scholar 

  44. Pretorius, C.J., du Plessis, M.C., Cilliers, C.B.: Towards an Artificial Neural Network-Based Simulator for Behavioural Evolution in Evolutionary Robotics. In: Proceedings of the 2009 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists, pp. 170–178. ACM (2009)

  45. Pretorius, C.J., du Plessis, M.C., Cilliers, C.B.: A Neural Network-Based Kinematic and Light-Perception Simulator for Simple Robotic Evolution. In: IEEE Congress on Evolutionary Computation (CEC), pp. 1–8 (2010)

  46. Pretorius, C.J., du Plessis, M.C., Cilliers, C.B.: A Neural Network Ultrasonic Sensor Simulator for Evolutionary Robotics. In: INFOCOMP 2012, the Second International Conference on Advanced Communications and Computation, pp. 54–61 (2012)

  47. Pretorius, C.J., du Plessis, M.C., Cilliers, C.B.: Simulating robots without conventional physics: a Neural Network approach. J. Intell. Robot. Syst. 71(3–4), 319–348 (2013)

  48. Pretorius, C.J., du Plessis, M.C., Gonsalves, J.W.: A Comparison of Neural Networks and Physics Models as Motion Simulators for Simple Robotic Evolution. In: IEEE Congress on Evolutionary Computation (CEC) (2014)

  49. Riedmiller, M., Braun, H.: A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm. In: IEEE International Conference on Neural Networks (1993)

  50. Schneider, S., Igel, C., Klaes, C., Dinse, H.R., Wiemer, J.C.: Evolutionary adaptation of nonlinear dynamical systems in computational neuroscience. Genet. Program Evolvable Mach. 5(2), 215–227 (2004)

    Article  Google Scholar 

  51. Stanley, K.O., Miikkulainen, R.: Evolving Neural Networks Through Augmenting Topologies. Evol. Comput. 10(2), 99–127 (2002)

    Article  Google Scholar 

  52. Sutton, R., Barto, A.: Reinforcement Learning: an Introduction. MIT Press, Cambridge (1998)

    Google Scholar 

  53. Whitley, D., Dominic, S., Das, R., Anderson, C.W.: Genetic Reinforcement Learning for neurocontrol problems. Mach. Learn. 13(2), 259–284 (1993)

    Article  Google Scholar 

  54. Wieland, A.: Evolving Neural Network Controllers for Unstable Systems. In: International Joint Conference on Neural Networks (1991)

  55. Yamamoto, Y.: NXTway-GS Model-Based Design - Control of self-balancing two-wheeled robot built with LEGO Mindstorms NXT. Online (2008). http://www.pages.drexel.edu/dml46/Tutorials/BalancingBot/files/NXTway-GS

  56. Yi, J., Yubazaki, N.: Stabilization fuzzy control of inverted pendulum systems. Artif. Intell. Eng. 14(2), 153–163 (2000)

    Article  Google Scholar 

  57. Zagal, J.C., Ruiz-del Solar, J.: Combining simulation and reality in Evolutionary Robotics. J. Intell. Robot. Syst. 50(1), 19–39 (2007)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics and Applied Mathematics, Nelson Mandela Metropolitan University, Port Elizabeth, South Africa

    Christiaan J. Pretorius & John W. Gonsalves

  2. Department of Computing Sciences, Nelson Mandela Metropolitan University, Port Elizabeth, South Africa

    Mathys C. du Plessis

Authors
  1. Christiaan J. Pretorius
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Mathys C. du Plessis
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. John W. Gonsalves
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Christiaan J. Pretorius.

Additional information

The financial assistance of the National Research Foundation (NRF) towards this research is hereby gratefully acknowledged (UID number: 79570). Opinions expressed and conclusions arrived at, are those of the authors and are not necessarily to be attributed to the NRF.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pretorius, C.J., du Plessis, M.C. & Gonsalves, J.W. Neuroevolution of Inverted Pendulum Control: A Comparative Study of Simulation Techniques. J Intell Robot Syst 86, 419–445 (2017). https://doi.org/10.1007/s10846-017-0465-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-017-0465-1

Keywords

Mathematics Subject Classification (2010)