Abstract
This paper presents an online model-free reinforcement learning based controller realized by approximate dynamic programming for a robotic knee as part of a human-machine system. Traditionally, prosthesis wearers’ gait performance is improved by manually tuning the impedance parameters. In this paper, we show that the parameter tuning problem can be formulated as an optimal control problem and thus solved by dynamic programming. Toward this goal, we constructed an quadratic instantaneous cost, which resulted in a value function that could be approximated by a neural network. The control policy is then solved by the least-squared method iteratively, a framework of which we refer to as approximate policy iteration. We performed extensive simulations based on prosthetic kinetics and human performance data extracted from real human subjects. Our results show that the proposed parameter tuning algorithm can be readily used for adaptive optimal tuning of prosthetic knee control parameters and the tuning process is time and sample efficient.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Jacobs, D.: From the ground up: building a passive dynamic walker model (2014)
Delp, S.L., et al.: Opensim: open-source software to create and analyze dynamic simulations of movement. IEEE Tran. Biomed. Eng. 54(11), 1940–1950 (2007)
Guo, W., Liu, F., Si, J., He, D., Harley, R., Mei, S.: Online supplementary ADP learning controller design and application to power system frequency control with large-scale wind energy integration. IEEE Trans. Neural Netw. Learn. Syst. 27(8), 1748–1761 (2016)
Guo, W., Si, J., Liu, F., Mei, S.: Policy approximation in policy iteration approximate dynamic programming for discrete-time nonlinear systems. IEEE Trans. Neural Netw. Learn. Syst. 29(7), 2794–2807 (2018)
Hogan, N.: Impedance control: an approach to manipulation: part iii applications. J. Dyn. Syst. Meas. Contr. 107(1), 17 (1985)
Howard, R.A.: Dynamic Programming and Markov Processes, 1st edn. MIT Press, Cambridge (1960)
Huang, H., Crouch, D.L., Liu, M., Sawicki, G.S., Wang, D.: A cyber expert system for auto-tuning powered prosthesis impedance control parameters. Ann. Biomed. Eng. 44(5), 1613–1624 (2016)
Liu, D., Wei, Q.: Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems. IEEE Trans. Neural Netw. Learn. Syst. 25(3), 621–634 (2014)
Liu, M., Zhang, F., Datseris, P., Huang, H.: Improving finite state impedance control of active-transfemoral prosthesis using dempster-shafer based state transition rules. J. Intell. Rob. Syst. Theor. Appl. 76(3–4), 461–474 (2014)
Si, J., Wang, Y.T.: Online learning control by association and reinforcement. IEEE Trans. Neural Netw. 12(2), 264–276 (2001)
Si, J., Barto, A.G., Powell, W.B., Wunsch, D. (eds.): Handbook of learning and approximate dynamic programming. Wiley, Piscataway (2004)
Wen, Y., Brandt, A., Liu, M., Huang, H., Si, J.: Comparing parallel and sequential control parameter tuning for a powered knee prosthesis joint department of biomedical engineering. In: IEEE International Conference on Systems, Man and Cybernetics, pp. 1716–1721 (2017)
Wen, Y., Liu, M., Si, J., Huang, H.: Adaptive control of powered transfemoral prostheses based on adaptive dynamic programming. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 5071–5074 (2016)
Wen, Y., Si, J., Gao, X., Huang, S., Huang, H.: A new powered lower limb prosthesis control framework based on adaptive dynamic programming. IEEE Trans. Neural Netw. Learn. Syst. 28(9), 2215–2220 (2017)
Acknowledgement
This work was partly supported by the National Science Foundation under grants #1406750, #1563454, #1563921, #1808752 and #1808898.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Gao, X., Wen, Y., Li, M., Si, J., Huang, H.(. (2019). Robotic Knee Parameter Tuning Using Approximate Policy Iteration. In: Sun, F., Liu, H., Hu, D. (eds) Cognitive Systems and Signal Processing. ICCSIP 2018. Communications in Computer and Information Science, vol 1005. Springer, Singapore. https://doi.org/10.1007/978-981-13-7983-3_49
Download citation
DOI: https://doi.org/10.1007/978-981-13-7983-3_49
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-7982-6
Online ISBN: 978-981-13-7983-3
eBook Packages: Computer ScienceComputer Science (R0)