Skip to main content
SpringerLink
Log in
Book cover

Evolution, Complexity and Artificial Life pp 107–122Cite as

Cumulative Learning Through Intrinsic Reinforcements

  • Chapter

Abstract

Building artificial agents able to autonomously learn new skills and to easily adapt in different and complex environments is an important goal for robotics and machine learning. We propose that providing reinforcement learning artificial agents with a learning signal that resembles the characteristic of the phasic activations of dopaminergic neurons would be an advancement in the development of more autonomous and versatile systems. In particular, we suggest that the particular composition of such a signal, determined by both extrinsic and intrinsic reinforcements, would be suitable to improve the implementation of cumulative learning in artificial agents. To validate our hypothesis we performed experiments with a simulated robotic system that has to learn different skills to obtain extrinsic rewards. We compare different versions of the system varying the composition of the learning signal and we show that the only system able to reach high performance in the task is the one that implements the learning signal suggested by our hypothesis.

An erratum to this chapter can be found at http://dx.doi.org/10.1007/978-3-642-37577-4_18

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD   54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Weng, J., McClelland, J., Pentland, A., Sporns, O., Stockman, I., Sur, M., Thelen, E.: Artificial intelligence. Autonomous mental development by robots and animals. Science 291(5504), 599–600 (2001)

    Google Scholar 

  2. Baldassarre, G., Mirolli, M.: What are the key open challenges for understanding autonomous cumulative learning of skills? Auton. Mental Develop. Newsl. 7(2), 2–9 (2010)

    Google Scholar 

  3. White, R.: Motivation reconsidered: The concept of competence. Psychol. Rev. 66, 297–333 (1959)

    Article  Google Scholar 

  4. Berlyne, D.: Conflict, Arousal and Curiosity. McGraw Hill, New York (1960)

    Google Scholar 

  5. Ryan, R.M., Deci, E.L.: Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemp. Educ. Psychol. 25(1), 54–67 (2000)

    Article  Google Scholar 

  6. Montgomery, K.: The role of the exploratory drive in learning. J. Comp. Psychol. 47(1), 60–64 (1954)

    Article  Google Scholar 

  7. Butler, R.A., Harlow, H.F.: Discrimination learning and learning sets to visual exploration incentives. J. Gen. Psychol. 57(2), 257–264 (1957)

    Article  Google Scholar 

  8. Hull, C.L.: Principles of behavior. Appleton-century-crofts, New York (1943)

    Google Scholar 

  9. Kish, G.B.: Learning when the onset of illumination is used as reinforcing stimulus. J. Comp. Physiol. Psychol. 48(4), 261–264 (1955)

    Article  Google Scholar 

  10. Glow, P., Winefield, A.: Response-contingent sensory change in a causally structured environment. Learn. Behav. 6, 1–18 (1978)

    Article  Google Scholar 

  11. Reed, P., Mitchell, C., Nokes, T.: Intrinsic reinforcing properties of putatively neutral stimuli in an instrumental two-lever discrimination task. Anim. Lear. Behav. 24, 38–45 (1996)

    Article  Google Scholar 

  12. Schmidhuber, J.: Curious model-building control system. In: Proceedings of International Joint Conference on Neural Networks, vol. 2, pp. 1458–1463. IEEE, Singapore (1991)

    Google Scholar 

  13. Huang, X., Weng, J.: Novelty and reinforcement learning in the value system of developmental robots. In: Prince, C., Demiris, Y., Marom, Y., Kozima, H., Balkenius, C. (eds.) Proceedings of the Second International Workshop on Epigenetic Robotics, vol. 94, pp. 47–55. Lund University (2002)

    Google Scholar 

  14. Oudeyer, P., Kaplan, F., Hafner, V.: Intrinsic motivation system for autonomous mental development. IEEE T. Evolut. Comput. 11, 703–713 (2007)

    Article  Google Scholar 

  15. Baranes, A., Oudeyer, P.Y.: Intrinsically motivated goal exloration for active motor learning in robots: A case study. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, Taipei, Taiwan (2010)

    Google Scholar 

  16. Barto, A., Singh, S., Chantanez, N.: Intrinsically motivated learning of hierarchical collections of skills. In: Proceedings of the Third International Conference on Developmental Learning (ICDL), pp. 112–119 (2004)

    Google Scholar 

  17. Schembri, M., Mirolli, M., Baldassarre, G.: Evolving internal reinforcers for an intrinsically motivated reinforcement-learning robot. In: Demiris, Y., Mareschal, D., Scassellati, B., Weng, J. (eds.) Proceedings of the 6th International Conference on Development and Learning, pp. E1–6. Imperial College, London (2007)

    Google Scholar 

  18. Oudeyer, P.Y., Kaplan, F.: What is intrinsic motivation? A typology of computational approaches. Front. Neurorobot. 1(1), 1–14 (2007)

    Google Scholar 

  19. Baldassarre, G., Mirolli, M. (eds.): Intrinsically Motivated Learning in Natural and Artificial Systems. Springer, Berlin (2013)

    Google Scholar 

  20. Santucci, V.G., Baldassarre, G., Mirolli, M.: Biological cumulative learning through intrinsic motivation: A simulated robotic study on the development of visually-guided reaching. In: Johansson, B., Sahin, E., Balkenius, C. (eds.) Proceedings of the Tenth International Conference on Epigenetic Robotics, pp. 121–128. Lund University Cognitive Studies, Lund (2010)

    Google Scholar 

  21. Wise, R.: Dopamine, learning and motivation. Nat. Rev. Neurosci. 5(6), 483–494 (2004)

    Article  MathSciNet  Google Scholar 

  22. Schultz, W.: Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 57, 87–115 (2006)

    Article  Google Scholar 

  23. Berridge, K.: The debate over dopamine’s role in reward: The case for incentive salience. Psychopharmacology. 191(3), 391–431 (2007)

    Article  Google Scholar 

  24. Romo, R., Schultz, W.: Dopamine neurons of the monkey midbrain: Contingencies of responses to active touch during self-initiated arm movements. J. Neurophysiol. 63(3), 592–606 (1990)

    Google Scholar 

  25. Ljungberg, T., Apicella, P., Schultz, W.: Responses of monkey midbrain dopamine neurons during delayed alternation performance. Brain Res. 567(2), 337–341 (1991)

    Article  Google Scholar 

  26. Schultz, W., Apicella, P., Ljumberg, T.: Responses of monkey dopamine neurons to reaward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993)

    Google Scholar 

  27. Mirenowicz, J., Schultz, W.: Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72(2), 1024–1027 (1994)

    Google Scholar 

  28. Ljungberg, T., Apicella, P., Schultz, W.: Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol. 67(1), 145–163 (1992)

    Google Scholar 

  29. Schultz, W.: Predictive reward signal of dopamine neurons. J. Neurophysiol. 80(1), 1–27 (1998)

    Google Scholar 

  30. Horvitz, J.C.: Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96(4), 651–656 (2000)

    Article  Google Scholar 

  31. Dommett, E., Coizet, V., Blaha, C.D., Martindale, J., Lefebvre, V., Walton, N., Mayhew, J.E.W., Overton, P.G., Redgrave, P.: How visual stimuli activate dopaminergic neurons at short latency. Science 307(5714), 1476–1479 (2005)

    Article  Google Scholar 

  32. Houk, J., Adams, J., Barto, A.: A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Houk, J.C., Davis, J.L., Beiser, D.G. (eds.) Models of Information Processing in the Basal Ganglia, pp. 249–270. MIT Press, Cambridge (1995)

    Google Scholar 

  33. Schultz, W., Dayan, P., Montague, P.R.: A neural substrate of prediction and reward. Science 275(5306), 1593–1599 (1997)

    Article  Google Scholar 

  34. Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)

    Google Scholar 

  35. Redgrave, P., Gurney, K.: The short-latency dopamine signal: A role in discovering novel actions? Nat. Rev. Neurosci. 7(12), 967–975 (2006)

    Article  Google Scholar 

  36. Redgrave, P., Vautrelle, N., Reynolds, J.N.J.: Functional properties of the basal ganglia’s re-entrant loop architecture: Selection and reinforcement. Neuroscience. 198, 138–151 (2011)

    Article  Google Scholar 

  37. Mirolli, M., Santucci, V.G., Baldassarre, G.: Phasic dopamine as a prediction error of intrinsic and extrinsic reinforcements driving both action acquisition and reward maximization: a simulated robotic study. Neural Networks 39, 40–51 (2013)

    Article  Google Scholar 

  38. Baldassarre, G.: What are intrinsic motivations? A biological perspective. In: Cangelosi, A., Triesch, J., Fasel, I., Rohlfing, K., Nori, F., Oudeyer, P.Y., Schlesinger, M., Nagai, Y. (eds.) Proceedings of the International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob-2011), pp. E1–8. IEEE, Piscataway (2011)

    Chapter  Google Scholar 

  39. Romanelli, P., Esposito, V., Schaal, D.W., Heit, G.: Somatotopy in the basal ganglia: Experimental and clinical evidence for segregated sensorimotor channels. Brain Res. Rev. 48(1), 112–128 (2005)

    Article  Google Scholar 

  40. Graybiel, A.M.: The basal ganglia: Learning new tricks and loving it. Curr. Opin. Neurobiol. 15(6), 638–644 (2005)

    Article  Google Scholar 

  41. Joel, D., Niv, Y., Ruppin, E.: Actor-critic models of the basal ganglia: New anatomical and computational perspectives. Neural Network 15(4–6), 535–547 (2002)

    Article  Google Scholar 

  42. Sutton, R.S., McAllester, D.A., Singh, S.P., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. In: NIPS, vol. 99, pp. 1057–1063 (1999)

    Google Scholar 

  43. Schultz, W.: Getting formal with dopamine and reward. Neuron 36(2), 241–263 (2002)

    Article  Google Scholar 

  44. Pouget, A., Snyder, L.H.: Computational approaches to sensorimotor transformations. Nat. Neurosci. 3(Suppl), 1192–1198 (2000)

    Article  Google Scholar 

  45. Buhmann, M.: Radial Basis Functions. Cambridge University Press, New York (2003)

    Book  MATH  Google Scholar 

  46. Sutton, R., Tanner, B.: Temporal-difference networks. Adv. Neural Inf. Process. Syst. 17, 1377–1348 (2005)

    Google Scholar 

  47. Singh, S., Lewis, R., Barto, A., Sorg, J.: Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE T. Auton. Mental Dev. 2(2), 70–82 (2010)

    Article  Google Scholar 

  48. Doya, K., Samejima, K., Katagiri, K., Kawato, M.: Multiple model-based reinforcement learning. Neural Comput. 14(6), 1347–1369 (2002)

    Article  MATH  Google Scholar 

  49. Caligiore, D., Mirolli, M., Parisi, D., Baldassarre, G.: A bioinspired hierarchical reinforcement learning architecture for modeling learning of multiple skills with continuous states and actions. In: Johansson, B., Sahin, E., Balkenius, C. (eds.) Proceedings of the Tenth International Conference on Epigenetic Robotics, pp. 27–34 (2010)

    Google Scholar 

  50. Schembri, M., Mirolli, M., Baldassarre, G.: Evolving childhood’s length and learning parameters in an intrinsically motivated reinforcement learning robot. In: Berthouze, L., Dhristiopher, G., Littman, M., Kozima, H., Balkenius, C. (eds.) Proceedings of the Seventh International Conference on Epigenetic Robotics, pp. 141–148. Lund University Cognitive Studies, Lund (2007)

    Google Scholar 

Download references

Acknowledgements

This research was supported by the European Community 7th Framework Programme (FP7/2007-2013), “Challenge 2 - Cognitive Systems, Interaction, Robotics”, grant agreement No. ICT-IP-231722, project “IM-CLeVeR - Intrinsically Motivated Cumulative Learning Versatile Robots”.

Author information

Authors and Affiliations

  1. Istituto di Scienze e Tecnologie della Cognizione (ISTC), Consiglio Nazionale delle Ricerche (CNR), Laboratory of Computational Embodied Neuroscience (LOCEN), Via San Martino della Battaglia 44, 00185, Roma, Italia

    Vieri G. Santucci, Gianluca Baldassarre & Marco Mirolli

  2. School of Computing and Mathematics, University of Plymouth, Plymouth, PL4 8AA, UK

    Vieri G. Santucci

Authors
  1. Vieri G. Santucci
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gianluca Baldassarre
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marco Mirolli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vieri G. Santucci .

Editor information

Editors and Affiliations

  1. Dept. of Information Engineering, University of Parma, Parma, Italy

    Stefano Cagnoni

  2. Consiglio Nazionale delle Ricerche, Istituto di Scienze e Tecnologie della Cognizione, Rome, Italy

    Marco Mirolli

  3. Facoltà di Scienze della Comunicazione, University of Modena and Reggio Emilia, Reggio Emilia, Italy

    Marco Villani

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Santucci, V.G., Baldassarre, G., Mirolli, M. (2014). Cumulative Learning Through Intrinsic Reinforcements. In: Cagnoni, S., Mirolli, M., Villani, M. (eds) Evolution, Complexity and Artificial Life. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-37577-4_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-37577-4_7

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-37576-7

  • Online ISBN: 978-3-642-37577-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation