Skip to main content
SpringerLink
Log in

Skill Learning for Robotic Insertion Based on One-shot Demonstration and Reinforcement Learning

  • Research Article
  • Published:
International Journal of Automation and Computing Aims and scope Submit manuscript

Abstract

In this paper, an efficient skill learning framework is proposed for robotic insertion, based on one-shot demonstration and reinforcement learning. First, the robot action is composed of two parts: expert action and refinement action. A force Jacobian matrix is calibrated with only one demonstration, based on which stable and safe expert action can be generated. The deep deterministic policy gradients (DDPG) method is employed to learn the refinement action, which aims to improve the assembly efficiency. Second, an episode-step exploration strategy is developed, which uses the expert action as a benchmark and adjusts the exploration intensity dynamically. A safety-efficiency reward function is designed for the compliant insertion. Third, to improve the adaptability with different components, a skill saving and selection mechanism is proposed. Several typical components are used to train the skill models. And the trained models and force Jacobian matrices are saved in a skill pool. Given a new component, the most appropriate model is selected from the skill pool according to the force Jacobian matrix and directly used to accomplish insertion tasks. Fourth, a simulation environment is established under the guidance of the force Jacobian matrix, which avoids tedious training process on real robotic systems. Simulation and experiments are conducted to validate the effectiveness of the proposed methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. S. Liu, D. Xu, D. P. Zhang, Z. T. Zhang. High precision automatic assembly based on microscopic vision and force information. IEEE Transactions on Automation Science and Engineering, vol. 13, no. 1, pp. 382–393, 2016. DOI: https://doi.org/10.1109/TASE.2014.2332543.

    Article  Google Scholar 

  2. J. Zhang, D. Xu, Z. T. Zhang, W. S. Zhang. Position/force hybrid control system for high precision aligning of small gripper to ring object. International Journal of Automation and Computing, vol. 10, no. 4, pp. 360–367, 2013. DOI: https://doi.org/10.1007/s11633-013-0732-y.

    Article  Google Scholar 

  3. F. B. Qin, D. Xu, D. P. Zhang, Y. Li. Robotic skill learning for precision assembly with microscopic vision and force feedback. IEEE/ASME Transactions on Mechatronics, vol. 24, no. 3, pp. 1117–1128, 2019. DOI: https://doi.org/10.1109/TMECH.2019.2909081.

    Article  Google Scholar 

  4. M. Armin, P. N. Roy, S. K. Das. A survey on modelling and compensation for hysteresis in high speed nanopositioning of AFMs: Observation and future recommendation. International Journal of Automation and Computing, vol. 17, no. 4, pp. 479–501, 2020. DOI: https://doi.org/10.1007/s11633-020-1225-4.

    Article  Google Scholar 

  5. K. G. Zhang, J. Xu, H. P. Chen, J. G. Zhao, K. Chen. Jamming analysis and force control for flexible dual peg-in-hole assembly. IEEE Transactions on Industrial Electronics, vol. 66, no. 3, pp. 1930–1939, 2019. DOI: https://doi.org/10.1109/TIE.2018.2838069.

    Article  Google Scholar 

  6. S. Liu, Y. F. Li, D. P. Xing. Sensing and control for simultaneous precision peg-in-hole assembly of multiple objects. IEEE Transactions on Automation Science and Engineering, vol. 17, no. 1, pp. 310–324, 2020. DOI: https://doi.org/10.1109/TASE.2019.2921224.

    Article  Google Scholar 

  7. F. Chen, F. Cannella, J. Huang, H. Sasaki, T. Fukuda. A study on error recovery search strategies of electronic connector mating for robotic fault-tolerant assembly. Journal of Intelligent & Robotic Systems, vol. 81, no. 2, pp. 257–271, 2016. DOI: https://doi.org/10.1007/s10846-015-0248-5.

    Article  Google Scholar 

  8. D. P. Xing, Y. Lv, S. Liu, D. Xu, F. F. Liu. Efficient insertion of multiple objects parallel connected by passive compliant mechanisms in precision assembly. IEEE Transactions on Industrial Informatics, vol. 15, no. 9, pp. 4878–4887, 2019. DOI: https://doi.org/10.1109/TII.2019.2897731.

    Article  Google Scholar 

  9. J. Takahashi, T. Fukukawa, T. Fukuda. Passive alignment principle for robotic assembly between a ring and a shaft with extremely narrow clearance. IEEE/ASME Transactions on Mechatronics, vol. 21, no. 1, pp. 196–204, 2016. DOI: https://doi.org/10.1109/TMECH.2015.2448639.

    Google Scholar 

  10. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, D. Hassabis. Human-level control through deep reinforcement learning. Nature, vol. 518, no. 7540, pp. 529–533, 2015. DOI: https://doi.org/10.1038/nature14236.

    Article  Google Scholar 

  11. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra. Continuous control with deep reinforcement learning. In Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico, 2016.

  12. J. L. Luo, E. Solowjow, C. T. Wen, J. A. Ojea, A. M. Agogino, A. Tamar, P. Abbeel. Reinforcement learning on variable impedance controller for high-precision robotic assembly. In Proceedings of the International Conference on Robotics and Automation, IEEE, Montreal, Canada, pp. 3080–3087, 2019. DOI: https://doi.org/10.1109/ICRA.2019.8793506.

    Google Scholar 

  13. T. Johannink, S. Bahl, A. Nair, J. L. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, S. Levine. Residual reinforcement learning for robot control. In Proceedings of International Conference on Robotics and Automation, IEEE, Montreal, Canada, pp. 6023–6029, 2019. DOI: https://doi.org/10.1109/ICRA.2019.8794127.

    Google Scholar 

  14. T. Inoue, G. De Magistris, A. Munawar, T. Yokoya, R. Tachibana. Deep reinforcement learning for high precision assembly tasks. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, Vancouver, Canada, pp. 819–825, 2017. DOI: https://doi.org/10.1109/IROS.2017.8202244.

    Google Scholar 

  15. F. M. Li, Q. Jiang, S. S. Zhang, M. Wei, R. Song. Robot skill acquisition in assembly process using deep reinforcement learning. Neurocomputing, vol. 345, pp. 92–102, 2019. DOI: https://doi.org/10.1016/j.neucom.2019.01.087.

    Article  Google Scholar 

  16. Y. X. Fan, J. L. Luo, M. Tomizuka. A learning framework for high precision industrial assembly. In Proceedings of International Conference on Robotics and Automation, IEEE, Montreal, Canada, pp. 811–817, 2019. DOI: https://doi.org/10.1109/ICRA.2019.8793659.

    Google Scholar 

  17. M. Vecerik, O. Sushkov, D. Barker, T. Rothörl, T. Hester, J. Scholz. A practical approach to insertion with variable socket position using deep reinforcement learning. In Proceedings of International Conference on Robotics and Automation, IEEE, Montreal, Canada, pp. 754–760, 2019. DOI: https://doi.org/10.1109/ICRA.2019.8794074.

    Google Scholar 

  18. J. L. Luo, E. Solowjow, C. G. Wen, J. A. Ojea, A. M. Agogino. Deep reinforcement learning for robotic assembly of mixed deformable and rigid objects. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, Madrid, Spain, pp. 2062–2069, 2018. DOI: https://doi.org/10.1109/IROS.2018.8594353.

    Google Scholar 

  19. G. Thomas, M. Chien, A. Tamar, J. A. Ojea, P. Abbeel. Learning robotic assembly from CAD. In Proceedings of IEEE International Conference on Robotics and Automation, IEEE, Brisbane, Australia, pp. 3524–3531, 2018. DOI: https://doi.org/10.1109/ICRA.2018.8460696.

    Google Scholar 

  20. Z. M. Hou, H. M. Dong, K. G. Zhang, Q. Gao, K. Chen, J. Xu. Knowledge-driven deep deterministic policy gradient for robotic multiple peg-in-hole assembly tasks. In Proceedings of IEEE International Conference on Robotics and Biomimetics, IEEE, Kuala Lumpur, Malaysia, pp. 256–261, 2018. DOI: https://doi.org/10.1109/ROBIO.2018.8665255.

    Google Scholar 

  21. J. Xu, Z. M. Hou, W. Wang, B. H. Xu, K. G. Zhang, K. Chen. Feedback deep deterministic policy gradient with fuzzy reward for robotic multiple peg-in-hole assembly tasks. IEEE Transactions on Industrial Informatics, vol. 15, no. 3, pp. 1658–1667, 2019. DOI: https://doi.org/10.1109/TII.2018.2868859.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Key Research and Development Program of China (No. 2018AAA0103005) and National Natural Science Foundation of China (No. 61873266).

Author information

Authors and Affiliations

  1. Research Center of Precision Sensing and Control, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Ying Li & De Xu

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100049, China

    Ying Li & De Xu

Authors
  1. Ying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. De Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to De Xu.

Additional information

Recommended by Associate Editor Nazim Mir-Nasiri

Colored figures are available in the online version at https://link.springer.com/journal/11633

Ying Li received the B.Sc. degree in control science and engineering from North China Electric Power University (Baoding), China in 2016. He is a Ph. D. degree candidate at Institute of Automation, Chinese Academy of Sciences (IACAS), China.

His research interests include visual measurement, visual control, micro-assembly and machine learning.

De Xu He received his B.Sc. degree in control science and engineering and M.Sc. degree in control science and engineering from Shandong University of Technology, China in 1985 and 1990, respectively, and received the Ph. D. degree in control science and engineering from Zhejiang University, China in 2001. He is a professor at the Institute of Automation, Chinese Academy of Sciences (IACAS), China.

His research interests include visual measurement, visual control, intelligent control, visual positioning, microscopic vision, and micro-assembly.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Xu, D. Skill Learning for Robotic Insertion Based on One-shot Demonstration and Reinforcement Learning. Int. J. Autom. Comput. 18, 457–467 (2021). https://doi.org/10.1007/s11633-021-1290-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-021-1290-3

Keywords

Search

Navigation